2024-03-27T05:59:27.477 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T05:59:27.477 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --name rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --verbose --archive /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580 -- /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/orig.config.yaml 2024-03-27T05:59:27.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T05:59:27.613 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580 branch: wip-rishabh-testing-20240326.131558 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7625580' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240326.131558 ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240326.131558 sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 6931 sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 sleep_before_teardown: 0 subset: 80/512 suite: fs suite_branch: wip-rishabh-testing-20240326.131558 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlC8b1Xb9cMub9U5aw87k8Okmsf9s9uwlDii5o/NuWDK9hG4g1SdEHKoONFWWqP1X63Vx69lC4WeZYVst7rfoI= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODXlBN77fBLoFuEWP82nY+5NNH+IIIOpJ3kkasqacKGnvNe23UPM31XQt76Zglt/8nIOdWIZthrOyatmFeLyeI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_05:27:11 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T05:59:27.614 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa; will attempt to use it 2024-03-27T05:59:27.614 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks 2024-03-27T05:59:27.615 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T05:59:27.616 INFO:teuthology.task.internal:Checking packages... 2024-03-27T05:59:27.639 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '155268c4e432a12433aa833f174f9fe3b1016ae0' 2024-03-27T05:59:27.639 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T05:59:27.639 INFO:teuthology.packaging:ref: None 2024-03-27T05:59:27.639 INFO:teuthology.packaging:tag: None 2024-03-27T05:59:27.639 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240326.131558 2024-03-27T05:59:27.639 INFO:teuthology.packaging:sha1: 155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T05:59:27.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240326.131558 2024-03-27T05:59:27.782 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2478.g155268c4 2024-03-27T05:59:27.784 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T05:59:27.790 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T05:59:27.790 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T05:59:27.859 INFO:teuthology.task.internal:Saving configuration 2024-03-27T05:59:27.877 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T05:59:27.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T05:59:27.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 05:54:25.389416', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlC8b1Xb9cMub9U5aw87k8Okmsf9s9uwlDii5o/NuWDK9hG4g1SdEHKoONFWWqP1X63Vx69lC4WeZYVst7rfoI='} 2024-03-27T05:59:27.973 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 05:54:25.387398', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODXlBN77fBLoFuEWP82nY+5NNH+IIIOpJ3kkasqacKGnvNe23UPM31XQt76Zglt/8nIOdWIZthrOyatmFeLyeI='} 2024-03-27T05:59:27.973 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T05:59:27.979 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-27T05:59:27.979 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-27T05:59:27.979 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T05:59:28.042 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7b5c72b160>, signals=[15]) 2024-03-27T05:59:28.042 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T05:59:28.048 INFO:teuthology.task.internal:Opening connections... 2024-03-27T05:59:28.048 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T05:59:28.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T05:59:28.141 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T05:59:28.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T05:59:28.231 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T05:59:28.238 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-03-27T05:59:28.255 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-03-27T05:59:28.256 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-03-27T05:59:28.313 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T05:59:28.314 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T05:59:28.315 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-03-27T05:59:28.335 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-03-27T05:59:28.352 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-03-27T05:59:28.352 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T05:59:28.411 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-03-27T05:59:28.412 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T05:59:28.412 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T05:59:28.412 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T05:59:28.412 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-03-27T05:59:28.434 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T05:59:28.444 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T05:59:28.451 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T05:59:28.451 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-03-27T05:59:28.454 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-03-27T05:59:28.469 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T05:59:28.475 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T05:59:28.475 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-03-27T05:59:28.510 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-03-27T05:59:28.745 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T05:59:28.800 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T05:59:28.816 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-27T05:59:28.816 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-27T05:59:28.816 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-27T05:59:28.816 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T05:59:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T05:59:28.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-27T05:59:29.147 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-27T05:59:29.147 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-27T05:59:29.147 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-27T05:59:29.164 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-361.el9.x86_64 2024-03-27T05:59:29.164 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-27T05:59:29.165 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-27T05:59:29.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum upgrade -y linux-firmware 2024-03-27T05:59:31.355 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-27T05:59:32.797 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-27T05:59:33.854 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-27T05:59:38.222 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-03-27T05:59:42.306 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-27T05:59:43.640 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-03-27T05:59:44.667 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-27T05:59:49.625 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-27T05:59:51.920 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-27T05:59:51.927 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 1 Package 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 375 M 2024-03-27T05:59:51.928 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-27T06:00:00.750 INFO:teuthology.orchestra.run.smithi132.stdout:linux-firmware-20240219-143.el9.noarch.rpm 43 MB/s | 375 MB 00:08 2024-03-27T06:00:00.751 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-27T06:00:00.752 INFO:teuthology.orchestra.run.smithi132.stdout:Total 42 MB/s | 375 MB 00:08 2024-03-27T06:00:00.752 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-27T06:00:00.829 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-27T06:00:00.829 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-27T06:00:03.301 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-27T06:00:03.301 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-27T06:00:09.617 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-27T06:00:09.851 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-27T06:00:09.985 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-27T06:00:11.359 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-27T06:00:11.359 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:00:11.762 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-27T06:00:12.046 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-27T06:00:12.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-27T06:00:12.270 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-27T06:00:12.271 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-27T06:00:12.273 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi132.front.sepia.ceph.com to complete... 2024-03-27T06:00:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27 06:00:23 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-27T06:00:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:FINISHED --2024-03-27 06:00:23-- 2024-03-27T06:00:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:Total wall clock time: 11s 2024-03-27T06:00:23.658 INFO:teuthology.orchestra.run.smithi132.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-27T06:00:23.661 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-27T06:00:23.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-27T06:00:25.603 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying... ######################################## 2024-03-27T06:00:25.739 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing... ######################################## 2024-03-27T06:00:25.820 INFO:teuthology.orchestra.run.smithi132.stdout:Updating / installing... 2024-03-27T06:00:33.308 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-27T06:00:49.824 INFO:teuthology.orchestra.run.smithi132.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T06:01:12.392 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-27T06:01:12.392 DEBUG:teuthology.orchestra.run.smithi132:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-27T06:01:12.462 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-27T06:01:12.463 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-27T06:01:12.464 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-27T06:01:12.465 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-27T06:01:12.466 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-27T06:01:12.467 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-27T06:01:12.468 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-27T06:01:12.469 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-27T06:01:12.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-27T06:01:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-27T06:01:12.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-27T06:01:12.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-27T06:01:12.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-27T06:01:12.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-27T06:01:12.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-27T06:01:12.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-27T06:01:12.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-27T06:01:12.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-27T06:01:12.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-27T06:01:12.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-27T06:01:12.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-27T06:01:12.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-27T06:01:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-27T06:01:12.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-27T06:01:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-27T06:01:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-27T06:01:12.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-27T06:01:12.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-27T06:01:12.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-27T06:01:12.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-27T06:01:12.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-27T06:01:12.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-27T06:01:12.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-27T06:01:12.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-27T06:01:12.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-27T06:01:12.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-27T06:01:12.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-27T06:01:12.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-27T06:01:12.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-27T06:01:12.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-27T06:01:12.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-27T06:01:12.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-27T06:01:12.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-27T06:01:12.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-27T06:01:12.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-27T06:01:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-27T06:01:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-27T06:01:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-27T06:01:12.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-27T06:01:12.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-27T06:01:12.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-27T06:01:12.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-27T06:01:12.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-27T06:01:12.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-27T06:01:12.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-27T06:01:12.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-27T06:01:12.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-27T06:01:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-27T06:01:12.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-27T06:01:12.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-27T06:01:12.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-27T06:01:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-27T06:01:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-27T06:01:12.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-27T06:01:12.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-27T06:01:12.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-27T06:01:12.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-27T06:01:12.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-27T06:01:12.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-27T06:01:12.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-27T06:01:12.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-27T06:01:12.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-27T06:01:12.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-27T06:01:12.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-27T06:01:12.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-27T06:01:12.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-27T06:01:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-27T06:01:12.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-27T06:01:12.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-27T06:01:12.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-27T06:01:12.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-27T06:01:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-27T06:01:12.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-27T06:01:12.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-27T06:01:12.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-27T06:01:12.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-27T06:01:12.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-27T06:01:12.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-27T06:01:12.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-27T06:01:12.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-27T06:01:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-27T06:01:12.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-27T06:01:12.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-27T06:01:12.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-27T06:01:12.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-27T06:01:12.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-27T06:01:12.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-27T06:01:12.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-27T06:01:12.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-27T06:01:12.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-27T06:01:12.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-27T06:01:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-27T06:01:12.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-27T06:01:12.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-27T06:01:12.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-27T06:01:12.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-27T06:01:12.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-27T06:01:12.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-27T06:01:12.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-27T06:01:12.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-27T06:01:12.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-27T06:01:12.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-27T06:01:12.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-27T06:01:12.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-27T06:01:12.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-27T06:01:12.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-27T06:01:12.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-27T06:01:12.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-27T06:01:12.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-27T06:01:12.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-27T06:01:12.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-27T06:01:12.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-27T06:01:12.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-27T06:01:12.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-27T06:01:12.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-27T06:01:12.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-27T06:01:12.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-27T06:01:12.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-27T06:01:12.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-27T06:01:12.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-27T06:01:12.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-27T06:01:12.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-27T06:01:12.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-27T06:01:12.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-27T06:01:12.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-27T06:01:12.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-27T06:01:12.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-27T06:01:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-27T06:01:12.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-27T06:01:12.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-27T06:01:12.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-27T06:01:12.606 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-27T06:01:12.606 DEBUG:teuthology.orchestra.run.smithi132:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-27T06:01:12.628 INFO:teuthology.orchestra.run.smithi132.stdout:postinstall scriptlet (using /bin/sh): 2024-03-27T06:01:12.628 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-27T06:01:12.628 INFO:teuthology.orchestra.run.smithi132.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-27T06:01:12.628 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:for file in vmlinuz System.map config; do 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout: fi 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:done 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-27T06:01:12.629 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-27T06:01:12.630 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-03-27T06:01:12.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-03-27T06:01:12.705 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T06:01:12.706 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:01:12.707 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T06:01:12.708 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T06:01:12.708 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 6.8.0-g1df0c7e8a235 2024-03-27T06:01:12.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T06:01:13.286 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-03-27T06:01:14.642 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T06:01:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-03-27T06:01:14.681 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-27T06:01:14.696 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.PAJY5ZE5n5 2024-03-27T06:01:14.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PAJY5ZE5n5 2024-03-27T06:01:14.765 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.PAJY5ZE5n5 2024-03-27T06:01:14.910 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.PAJY5ZE5n5 is 6KB 2024-03-27T06:01:14.959 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.PAJY5ZE5n5 2024-03-27T06:01:14.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T06:01:15.042 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T06:01:15.042 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T06:01:15.042 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T06:01:15.042 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-27T06:01:15.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-27T06:01:15.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-03-27T06:01:45.199 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-27T06:01:45.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:01:45.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T06:02:03.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-03-27T06:02:12.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:02:12.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T06:03:12.684 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T06:03:24.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:03:24.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T06:03:25.128 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-27T06:03:25.790 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:03:25.791 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-27T06:03:25.791 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-27T06:03:25.843 INFO:teuthology.orchestra.run.smithi132.stdout:6.8.0-g1df0c7e8a235 2024-03-27T06:03:25.843 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-27T06:03:25.843 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T06:03:25.843 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T06:03:26.846 DEBUG:teuthology.parallel:result is None 2024-03-27T06:03:26.847 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T06:03:26.859 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T06:03:26.859 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T06:03:26.866 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T06:03:26.914 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T06:03:26.921 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T06:03:26.931 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T06:03:26.931 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T06:03:26.938 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T06:03:27.057 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T06:03:27.064 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T06:03:27.064 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T06:03:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:03:27.080 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T06:03:27.088 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T06:03:27.197 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T06:03:27.214 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T06:03:27.231 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T06:03:27.298 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T06:03:27.303 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T06:03:27.310 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T06:03:27.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T06:03:27.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T06:03:27.406 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T06:03:27.420 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T06:03:27.420 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T06:03:27.425 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T06:03:27.472 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T06:03:27.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T06:03:27.657 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T06:03:27.740 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T06:03:27.837 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:03:27.837 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T06:03:27.926 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T06:03:27.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T06:03:28.030 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T06:03:28.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T06:03:28.118 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:03:28.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T06:03:28.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-03-27T06:03:28.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-03-27T06:03:28.275 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T06:03:28.317 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T06:03:28.751 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T06:03:28.759 INFO:teuthology.task.internal:Starting timer... 2024-03-27T06:03:28.759 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T06:03:28.818 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T06:03:28.844 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-27T06:03:28.845 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-03-27T06:03:28.974 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-03-27T06:03:28.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-03-27T06:03:29.038 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-03-27T06:03:29.040 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T06:03:29.040 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-03-27T06:03:29.101 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-03-27T06:03:29.102 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-03-27T06:03:29.126 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-03-27T06:03:29.127 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-03-27T06:03:29.127 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T06:03:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:03:29.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T06:03:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:03:29.246 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T06:03:29.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-03-27T06:03:29.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-03-27T06:03:29.387 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T06:03:29.398 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T06:03:29.418 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T06:03:29.419 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T06:11:00.288 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-03-27T06:11:00.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:11:00.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T06:11:00.445 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-27T06:11:00.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-27T06:11:00.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-27T06:11:00.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T06:11:00.693 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-27T06:11:00.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-27T06:11:00.762 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T06:11:00.774 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T06:11:00.774 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T06:11:00.774 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T06:11:00.782 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T06:11:00.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T06:11:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T06:11:00.853 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T06:11:00.876 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T06:11:00.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-03-27T06:11:00.896 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-03-27T06:11:00.912 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T06:11:00.924 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T06:11:00.929 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T06:11:00.973 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-27T06:11:00.976 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T06:11:00.976 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-27T06:11:00.976 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:00.977 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:00.977 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:00.977 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:01.019 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-03-27T06:11:01.063 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-03-27T06:11:01.114 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T06:11:01.167 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T06:11:01.300 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:11:01.310 INFO:teuthology.run_tasks:Running task install... 2024-03-27T06:11:01.318 DEBUG:teuthology.task.install:project ceph 2024-03-27T06:11:01.319 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'}} 2024-03-27T06:11:01.319 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'} 2024-03-27T06:11:01.319 INFO:teuthology.task.install:Using flavor: default 2024-03-27T06:11:01.327 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T06:11:01.327 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-27T06:11:01.328 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0', 'tag': None, 'wait_for_package': False} 2024-03-27T06:11:01.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:11:01.333 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0', 'tag': None, 'wait_for_package': False} 2024-03-27T06:11:01.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:11:01.612 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/ 2024-03-27T06:11:01.612 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2478.g155268c4 2024-03-27T06:11:01.618 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/ 2024-03-27T06:11:01.618 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2478.g155268c4 2024-03-27T06:11:01.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T06:11:01.731 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:11:01.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T06:11:01.822 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240326.131558/155268c4e432a12433aa833f174f9fe3b1016ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T06:11:01.822 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:11:01.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T06:11:01.834 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-27T06:11:01.834 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/155268c4e432a12433aa833f174f9fe3b1016ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T06:11:01.854 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-27T06:11:01.854 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/155268c4e432a12433aa833f174f9fe3b1016ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T06:11:01.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T06:11:01.971 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T06:11:02.005 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T06:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-03-27T06:11:02.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-03-27T06:11:02.103 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T06:11:02.190 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-03-27T06:11:02.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-03-27T06:11:02.307 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-03-27T06:11:02.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-27T06:11:02.560 INFO:teuthology.orchestra.run.smithi132.stdout:74 files removed 2024-03-27T06:11:02.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-27T06:11:03.060 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-03-27T06:11:03.494 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-03-27T06:11:03.569 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-03-27T06:11:03.911 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-27T06:11:04.086 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-03-27T06:11:04.364 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-27T06:11:04.545 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-27T06:11:05.182 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-27T06:11:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-27T06:11:07.041 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-27T06:11:08.795 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-03-27T06:11:10.738 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-27T06:11:15.072 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-27T06:11:15.229 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 907 kB/s | 8.0 MB 00:09 2024-03-27T06:11:16.639 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-27T06:11:17.363 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-27T06:11:17.816 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-27T06:11:23.632 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-27T06:11:26.260 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-27T06:11:26.262 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-27T06:11:26.283 INFO:teuthology.orchestra.run.smithi132.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-27T06:11:26.285 INFO:teuthology.orchestra.run.smithi132.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-27T06:11:26.286 INFO:teuthology.orchestra.run.smithi132.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-27T06:11:26.288 INFO:teuthology.orchestra.run.smithi132.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-27T06:11:26.290 INFO:teuthology.orchestra.run.smithi132.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-27T06:11:26.294 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-27T06:11:26.295 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-27T06:11:26.389 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-2478.g155268c4.el9 ceph 6.6 k 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-2478.g155268c4.el9 ceph 5.4 M 2024-03-27T06:11:26.412 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 890 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2478.g155268c4.el9 ceph 145 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-2478.g155268c4.el9 ceph 1.5 M 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 141 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 3.5 M 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 7.4 M 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 49 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-2478.g155268c4.el9 ceph 9.8 M 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-2478.g155268c4.el9 ceph 48 M 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 264 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 762 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror x86_64 2:19.0.0-2478.g155268c4.el9 ceph 221 k 2024-03-27T06:11:26.413 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell x86_64 2:19.0.0-2478.g155268c4.el9 ceph 27 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 18 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-2478.g155268c4.el9 ceph 32 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 746 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-2478.g155268c4.el9 ceph 127 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-27T06:11:26.414 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-2478.g155268c4.el9 ceph 162 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-2478.g155268c4.el9 ceph 320 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 298 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-2478.g155268c4.el9 ceph 99 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 85 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.1 M 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 170 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-27T06:11:26.415 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.4 M 2024-03-27T06:11:26.416 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.1 M 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-27T06:11:26.417 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-2478.g155268c4.el9 ceph 21 M 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 34 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-2478.g155268c4.el9 ceph 2.4 M 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 249 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-2478.g155268c4.el9 ceph 4.7 M 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 17 M 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 16 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-2478.g155268c4.el9 ceph 25 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T06:11:26.418 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-2478.g155268c4.el9 ceph 163 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 501 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T06:11:26.419 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 5.1 M 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T06:11:26.420 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 45 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-2478.g155268c4.el9 ceph 134 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T06:11:26.421 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T06:11:26.422 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T06:11:26.423 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T06:11:26.424 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-27T06:11:26.425 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-03-27T06:11:26.426 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout:Install 150 Packages 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 24 Packages 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:11:26.427 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 227 M 2024-03-27T06:11:26.428 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-27T06:11:26.909 INFO:teuthology.orchestra.run.smithi132.stdout:(1/174): ceph-19.0.0-2478.g155268c4.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-03-27T06:11:27.659 INFO:teuthology.orchestra.run.smithi132.stdout:(2/174): ceph-fuse-19.0.0-2478.g155268c4.el9.x8 1.2 MB/s | 890 kB 00:00 2024-03-27T06:11:28.009 INFO:teuthology.orchestra.run.smithi132.stdout:(3/174): ceph-common-19.0.0-2478.g155268c4.el9. 15 MB/s | 21 MB 00:01 2024-03-27T06:11:28.034 INFO:teuthology.orchestra.run.smithi132.stdout:(4/174): ceph-immutable-object-cache-19.0.0-247 386 kB/s | 145 kB 00:00 2024-03-27T06:11:28.193 INFO:teuthology.orchestra.run.smithi132.stdout:(5/174): ceph-mds-19.0.0-2478.g155268c4.el9.x86 13 MB/s | 2.4 MB 00:00 2024-03-27T06:11:28.485 INFO:teuthology.orchestra.run.smithi132.stdout:(6/174): ceph-mon-19.0.0-2478.g155268c4.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-27T06:11:28.861 INFO:teuthology.orchestra.run.smithi132.stdout:(7/174): ceph-mgr-19.0.0-2478.g155268c4.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-03-27T06:11:29.096 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 760 kB/s | 6.1 MB 00:08 2024-03-27T06:11:29.194 INFO:teuthology.orchestra.run.smithi132.stdout:(8/174): ceph-base-19.0.0-2478.g155268c4.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-03-27T06:11:29.461 INFO:teuthology.orchestra.run.smithi132.stdout:(9/174): ceph-osd-19.0.0-2478.g155268c4.el9.x86 17 MB/s | 17 MB 00:00 2024-03-27T06:11:29.486 INFO:teuthology.orchestra.run.smithi132.stdout:(10/174): ceph-selinux-19.0.0-2478.g155268c4.el 86 kB/s | 25 kB 00:00 2024-03-27T06:11:29.661 INFO:teuthology.orchestra.run.smithi132.stdout:(11/174): cephfs-mirror-19.0.0-2478.g155268c4.e 1.2 MB/s | 221 kB 00:00 2024-03-27T06:11:29.770 INFO:teuthology.orchestra.run.smithi132.stdout:(12/174): cephfs-shell-19.0.0-2478.g155268c4.el 252 kB/s | 27 kB 00:00 2024-03-27T06:11:29.878 INFO:teuthology.orchestra.run.smithi132.stdout:(13/174): libcephfs-devel-19.0.0-2478.g155268c4 295 kB/s | 32 kB 00:00 2024-03-27T06:11:30.320 INFO:teuthology.orchestra.run.smithi132.stdout:(14/174): libcephfs2-19.0.0-2478.g155268c4.el9. 1.7 MB/s | 746 kB 00:00 2024-03-27T06:11:30.437 INFO:teuthology.orchestra.run.smithi132.stdout:(15/174): libcephsqlite-19.0.0-2478.g155268c4.e 1.4 MB/s | 163 kB 00:00 2024-03-27T06:11:30.620 INFO:teuthology.orchestra.run.smithi132.stdout:(16/174): librados-devel-19.0.0-2478.g155268c4. 700 kB/s | 127 kB 00:00 2024-03-27T06:11:30.655 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-27T06:11:30.905 INFO:teuthology.orchestra.run.smithi132.stdout:(17/174): libradosstriper1-19.0.0-2478.g155268c 1.7 MB/s | 501 kB 00:00 2024-03-27T06:11:31.130 INFO:teuthology.orchestra.run.smithi132.stdout:(18/174): ceph-radosgw-19.0.0-2478.g155268c4.el 4.3 MB/s | 9.8 MB 00:02 2024-03-27T06:11:31.197 INFO:teuthology.orchestra.run.smithi132.stdout:(19/174): python3-ceph-argparse-19.0.0-2478.g15 688 kB/s | 45 kB 00:00 2024-03-27T06:11:31.272 INFO:teuthology.orchestra.run.smithi132.stdout:(20/174): python3-ceph-common-19.0.0-2478.g1552 1.8 MB/s | 134 kB 00:00 2024-03-27T06:11:31.401 INFO:teuthology.orchestra.run.smithi132.stdout:(21/174): python3-cephfs-19.0.0-2478.g155268c4. 1.2 MB/s | 162 kB 00:00 2024-03-27T06:11:31.539 INFO:teuthology.orchestra.run.smithi132.stdout:(22/174): python3-rados-19.0.0-2478.g155268c4.e 2.3 MB/s | 320 kB 00:00 2024-03-27T06:11:32.056 INFO:teuthology.orchestra.run.smithi132.stdout:(23/174): ceph-test-19.0.0-2478.g155268c4.el9.x 18 MB/s | 48 MB 00:02 2024-03-27T06:11:32.090 INFO:teuthology.orchestra.run.smithi132.stdout:(24/174): python3-rbd-19.0.0-2478.g155268c4.el9 542 kB/s | 298 kB 00:00 2024-03-27T06:11:32.115 INFO:teuthology.orchestra.run.smithi132.stdout:(25/174): python3-rgw-19.0.0-2478.g155268c4.el9 1.7 MB/s | 99 kB 00:00 2024-03-27T06:11:32.174 INFO:teuthology.orchestra.run.smithi132.stdout:(26/174): rbd-fuse-19.0.0-2478.g155268c4.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-27T06:11:32.316 INFO:teuthology.orchestra.run.smithi132.stdout:(27/174): rbd-nbd-19.0.0-2478.g155268c4.el9.x86 1.2 MB/s | 170 kB 00:00 2024-03-27T06:11:32.391 INFO:teuthology.orchestra.run.smithi132.stdout:(28/174): rbd-mirror-19.0.0-2478.g155268c4.el9. 11 MB/s | 3.1 MB 00:00 2024-03-27T06:11:32.408 INFO:teuthology.orchestra.run.smithi132.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2478.g 373 kB/s | 34 kB 00:00 2024-03-27T06:11:32.441 INFO:teuthology.orchestra.run.smithi132.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2478.g155268c 2.8 MB/s | 141 kB 00:00 2024-03-27T06:11:32.551 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-27T06:11:32.909 INFO:teuthology.orchestra.run.smithi132.stdout:(31/174): librgw2-19.0.0-2478.g155268c4.el9.x86 2.6 MB/s | 5.1 MB 00:02 2024-03-27T06:11:33.042 INFO:teuthology.orchestra.run.smithi132.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-27T06:11:33.101 INFO:teuthology.orchestra.run.smithi132.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2478.g15 1.3 MB/s | 249 kB 00:00 2024-03-27T06:11:33.126 INFO:teuthology.orchestra.run.smithi132.stdout:(34/174): ceph-mgr-rook-19.0.0-2478.g155268c4.e 598 kB/s | 49 kB 00:00 2024-03-27T06:11:33.209 INFO:teuthology.orchestra.run.smithi132.stdout:(35/174): ceph-mgr-dashboard-19.0.0-2478.g15526 4.4 MB/s | 3.5 MB 00:00 2024-03-27T06:11:33.235 INFO:teuthology.orchestra.run.smithi132.stdout:(36/174): ceph-prometheus-alerts-19.0.0-2478.g1 124 kB/s | 16 kB 00:00 2024-03-27T06:11:33.268 INFO:teuthology.orchestra.run.smithi132.stdout:(37/174): ceph-volume-19.0.0-2478.g155268c4.el9 1.8 MB/s | 264 kB 00:00 2024-03-27T06:11:33.294 INFO:teuthology.orchestra.run.smithi132.stdout:(38/174): cephfs-top-19.0.0-2478.g155268c4.el9. 315 kB/s | 18 kB 00:00 2024-03-27T06:11:33.411 INFO:teuthology.orchestra.run.smithi132.stdout:(39/174): cephadm-19.0.0-2478.g155268c4.el9.noa 3.7 MB/s | 762 kB 00:00 2024-03-27T06:11:33.586 INFO:teuthology.orchestra.run.smithi132.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-27T06:11:33.646 INFO:teuthology.orchestra.run.smithi132.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-03-27T06:11:33.670 INFO:teuthology.orchestra.run.smithi132.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 528 kB/s | 106 kB 00:00 2024-03-27T06:11:33.703 INFO:teuthology.orchestra.run.smithi132.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 307 kB 00:00 2024-03-27T06:11:33.728 INFO:teuthology.orchestra.run.smithi132.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 32 kB 00:00 2024-03-27T06:11:33.937 INFO:teuthology.orchestra.run.smithi132.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-27T06:11:33.962 INFO:teuthology.orchestra.run.smithi132.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 44 kB/s | 10 kB 00:00 2024-03-27T06:11:33.988 INFO:teuthology.orchestra.run.smithi132.stdout:(47/174): flexiblas-openblas-openmp-3.0.4-8.el9 67 kB/s | 17 kB 00:00 2024-03-27T06:11:34.013 INFO:teuthology.orchestra.run.smithi132.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 44 kB 00:00 2024-03-27T06:11:34.046 INFO:teuthology.orchestra.run.smithi132.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-27T06:11:34.080 INFO:teuthology.orchestra.run.smithi132.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-27T06:11:34.122 INFO:teuthology.orchestra.run.smithi132.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-03-27T06:11:34.147 INFO:teuthology.orchestra.run.smithi132.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 758 kB/s | 46 kB 00:00 2024-03-27T06:11:34.181 INFO:teuthology.orchestra.run.smithi132.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 282 kB 00:00 2024-03-27T06:11:34.214 INFO:teuthology.orchestra.run.smithi132.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 187 kB 00:00 2024-03-27T06:11:34.264 INFO:teuthology.orchestra.run.smithi132.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 34 kB 00:00 2024-03-27T06:11:34.307 INFO:teuthology.orchestra.run.smithi132.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-27T06:11:34.473 INFO:teuthology.orchestra.run.smithi132.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-27T06:11:34.524 INFO:teuthology.orchestra.run.smithi132.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.1 MB/s | 1.0 MB 00:00 2024-03-27T06:11:34.549 INFO:teuthology.orchestra.run.smithi132.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 340 kB/s | 78 kB 00:00 2024-03-27T06:11:34.766 INFO:teuthology.orchestra.run.smithi132.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-27T06:11:34.791 INFO:teuthology.orchestra.run.smithi132.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 107 kB/s | 24 kB 00:00 2024-03-27T06:11:34.816 INFO:teuthology.orchestra.run.smithi132.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 71 kB/s | 18 kB 00:00 2024-03-27T06:11:34.850 INFO:teuthology.orchestra.run.smithi132.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 997 kB/s | 32 kB 00:00 2024-03-27T06:11:34.892 INFO:teuthology.orchestra.run.smithi132.stdout:(64/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-27T06:11:35.067 INFO:teuthology.orchestra.run.smithi132.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-27T06:11:35.101 INFO:teuthology.orchestra.run.smithi132.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 917 kB/s | 217 kB 00:00 2024-03-27T06:11:35.126 INFO:teuthology.orchestra.run.smithi132.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 794 kB/s | 174 kB 00:00 2024-03-27T06:11:35.151 INFO:teuthology.orchestra.run.smithi132.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 863 kB/s | 69 kB 00:00 2024-03-27T06:11:35.176 INFO:teuthology.orchestra.run.smithi132.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 41 kB 00:00 2024-03-27T06:11:35.210 INFO:teuthology.orchestra.run.smithi132.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-27T06:11:35.235 INFO:teuthology.orchestra.run.smithi132.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-27T06:11:35.260 INFO:teuthology.orchestra.run.smithi132.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 989 kB/s | 47 kB 00:00 2024-03-27T06:11:35.294 INFO:teuthology.orchestra.run.smithi132.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-27T06:11:35.319 INFO:teuthology.orchestra.run.smithi132.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 518 kB/s | 43 kB 00:00 2024-03-27T06:11:35.544 INFO:teuthology.orchestra.run.smithi132.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 183 kB/s | 39 kB 00:00 2024-03-27T06:11:35.569 INFO:teuthology.orchestra.run.smithi132.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 174 kB/s | 41 kB 00:00 2024-03-27T06:11:35.778 INFO:teuthology.orchestra.run.smithi132.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-27T06:11:35.803 INFO:teuthology.orchestra.run.smithi132.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 252 kB/s | 56 kB 00:00 2024-03-27T06:11:35.862 INFO:teuthology.orchestra.run.smithi132.stdout:(79/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 300 kB 00:00 2024-03-27T06:11:35.887 INFO:teuthology.orchestra.run.smithi132.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 584 kB/s | 63 kB 00:00 2024-03-27T06:11:35.912 INFO:teuthology.orchestra.run.smithi132.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 418 kB/s | 44 kB 00:00 2024-03-27T06:11:35.946 INFO:teuthology.orchestra.run.smithi132.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-27T06:11:35.980 INFO:teuthology.orchestra.run.smithi132.stdout:(83/174): fuse-2.9.9-15.el9.x86_64.rpm 865 kB/s | 79 kB 00:00 2024-03-27T06:11:36.022 INFO:teuthology.orchestra.run.smithi132.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-27T06:11:36.047 INFO:teuthology.orchestra.run.smithi132.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-27T06:11:36.072 INFO:teuthology.orchestra.run.smithi132.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-27T06:11:36.097 INFO:teuthology.orchestra.run.smithi132.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-03-27T06:11:36.306 INFO:teuthology.orchestra.run.smithi132.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 733 kB/s | 187 kB 00:00 2024-03-27T06:11:36.331 INFO:teuthology.orchestra.run.smithi132.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 840 kB/s | 195 kB 00:00 2024-03-27T06:11:36.406 INFO:teuthology.orchestra.run.smithi132.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-27T06:11:36.431 INFO:teuthology.orchestra.run.smithi132.stdout:(91/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-27T06:11:36.707 INFO:teuthology.orchestra.run.smithi132.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 57 kB/s | 21 kB 00:00 2024-03-27T06:11:36.732 INFO:teuthology.orchestra.run.smithi132.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 78 kB/s | 25 kB 00:00 2024-03-27T06:11:37.382 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-27T06:11:37.574 INFO:teuthology.orchestra.run.smithi132.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 1.1 MB/s | 911 kB 00:00 2024-03-27T06:11:38.108 INFO:teuthology.orchestra.run.smithi132.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 250 kB/s | 133 kB 00:00 2024-03-27T06:11:38.483 INFO:teuthology.orchestra.run.smithi132.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 2.0 MB/s | 4.1 MB 00:02 2024-03-27T06:11:39.000 INFO:teuthology.orchestra.run.smithi132.stdout:(97/174): postgresql-13.14-1.el9.x86_64.rpm 709 kB/s | 1.6 MB 00:02 2024-03-27T06:11:39.025 INFO:teuthology.orchestra.run.smithi132.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 453 kB/s | 245 kB 00:00 2024-03-27T06:11:39.525 INFO:teuthology.orchestra.run.smithi132.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 473 kB/s | 248 kB 00:00 2024-03-27T06:11:39.609 INFO:teuthology.orchestra.run.smithi132.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 379 kB/s | 221 kB 00:00 2024-03-27T06:11:39.777 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-27T06:11:39.778 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-27T06:11:39.791 INFO:teuthology.orchestra.run.smithi134.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-27T06:11:39.792 INFO:teuthology.orchestra.run.smithi134.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-27T06:11:39.793 INFO:teuthology.orchestra.run.smithi134.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-27T06:11:39.793 INFO:teuthology.orchestra.run.smithi134.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-27T06:11:39.795 INFO:teuthology.orchestra.run.smithi134.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-27T06:11:39.798 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-27T06:11:39.799 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-27T06:11:39.817 INFO:teuthology.orchestra.run.smithi132.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 123 kB/s | 36 kB 00:00 2024-03-27T06:11:39.879 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-2478.g155268c4.el9 ceph 6.6 k 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-2478.g155268c4.el9 ceph 5.4 M 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 890 k 2024-03-27T06:11:39.905 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2478.g155268c4.el9 ceph 145 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-2478.g155268c4.el9 ceph 1.5 M 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 141 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 3.5 M 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 7.4 M 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 49 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-2478.g155268c4.el9 ceph 9.8 M 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-2478.g155268c4.el9 ceph 48 M 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 264 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 762 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror x86_64 2:19.0.0-2478.g155268c4.el9 ceph 221 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell x86_64 2:19.0.0-2478.g155268c4.el9 ceph 27 k 2024-03-27T06:11:39.906 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 18 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-2478.g155268c4.el9 ceph 32 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 746 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-2478.g155268c4.el9 ceph 127 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-27T06:11:39.907 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-2478.g155268c4.el9 ceph 162 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-2478.g155268c4.el9 ceph 320 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 298 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-2478.g155268c4.el9 ceph 99 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 85 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.1 M 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 170 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-27T06:11:39.908 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.4 M 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 3.1 M 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-27T06:11:39.909 INFO:teuthology.orchestra.run.smithi134.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-2478.g155268c4.el9 ceph 21 M 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 34 k 2024-03-27T06:11:39.910 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-2478.g155268c4.el9 ceph 2.4 M 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 249 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-2478.g155268c4.el9 ceph 4.7 M 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-2478.g155268c4.el9 ceph 17 M 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2478.g155268c4.el9 ceph-noarch 16 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-2478.g155268c4.el9 ceph 25 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T06:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-2478.g155268c4.el9 ceph 163 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 501 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-2478.g155268c4.el9 ceph 5.1 M 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T06:11:39.912 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T06:11:39.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-2478.g155268c4.el9 ceph 45 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-2478.g155268c4.el9 ceph 134 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T06:11:39.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T06:11:39.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T06:11:39.916 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T06:11:39.917 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T06:11:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T06:11:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout:Install 150 Packages 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 24 Packages 2024-03-27T06:11:39.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:11:39.921 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 227 M 2024-03-27T06:11:39.921 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-27T06:11:39.960 INFO:teuthology.orchestra.run.smithi132.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 136 kB/s | 47 kB 00:00 2024-03-27T06:11:40.060 INFO:teuthology.orchestra.run.smithi132.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 225 kB/s | 22 kB 00:00 2024-03-27T06:11:40.277 INFO:teuthology.orchestra.run.smithi132.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 204 kB/s | 94 kB 00:00 2024-03-27T06:11:40.361 INFO:teuthology.orchestra.run.smithi134.stdout:(1/174): ceph-19.0.0-2478.g155268c4.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-27T06:11:40.460 INFO:teuthology.orchestra.run.smithi132.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-27T06:11:40.544 INFO:teuthology.orchestra.run.smithi134.stdout:(2/174): ceph-fuse-19.0.0-2478.g155268c4.el9.x8 4.8 MB/s | 890 kB 00:00 2024-03-27T06:11:40.556 INFO:teuthology.orchestra.run.smithi132.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-27T06:11:40.603 INFO:teuthology.orchestra.run.smithi132.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 435 kB/s | 19 kB 00:00 2024-03-27T06:11:40.603 INFO:teuthology.orchestra.run.smithi134.stdout:(3/174): ceph-immutable-object-cache-19.0.0-247 2.4 MB/s | 145 kB 00:00 2024-03-27T06:11:40.686 INFO:teuthology.orchestra.run.smithi132.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-27T06:11:40.728 INFO:teuthology.orchestra.run.smithi132.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 636 kB/s | 26 kB 00:00 2024-03-27T06:11:40.778 INFO:teuthology.orchestra.run.smithi132.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 988 kB/s | 49 kB 00:00 2024-03-27T06:11:40.870 INFO:teuthology.orchestra.run.smithi132.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-27T06:11:40.896 INFO:teuthology.orchestra.run.smithi132.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2024-03-27T06:11:40.952 INFO:teuthology.orchestra.run.smithi132.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-27T06:11:40.953 INFO:teuthology.orchestra.run.smithi134.stdout:(4/174): ceph-mds-19.0.0-2478.g155268c4.el9.x86 6.8 MB/s | 2.4 MB 00:00 2024-03-27T06:11:41.022 INFO:teuthology.orchestra.run.smithi132.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 7.8 MB/s | 548 kB 00:00 2024-03-27T06:11:41.072 INFO:teuthology.orchestra.run.smithi132.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-27T06:11:41.097 INFO:teuthology.orchestra.run.smithi132.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 586 kB/s | 43 kB 00:00 2024-03-27T06:11:41.122 INFO:teuthology.orchestra.run.smithi132.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 649 kB/s | 32 kB 00:00 2024-03-27T06:11:41.148 INFO:teuthology.orchestra.run.smithi132.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2024-03-27T06:11:41.154 INFO:teuthology.orchestra.run.smithi134.stdout:(5/174): ceph-base-19.0.0-2478.g155268c4.el9.x8 5.6 MB/s | 5.4 MB 00:00 2024-03-27T06:11:41.181 INFO:teuthology.orchestra.run.smithi132.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-27T06:11:41.196 INFO:teuthology.orchestra.run.smithi134.stdout:(6/174): ceph-mgr-19.0.0-2478.g155268c4.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-27T06:11:41.232 INFO:teuthology.orchestra.run.smithi132.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-27T06:11:41.265 INFO:teuthology.orchestra.run.smithi132.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-27T06:11:41.290 INFO:teuthology.orchestra.run.smithi132.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 610 kB/s | 35 kB 00:00 2024-03-27T06:11:41.349 INFO:teuthology.orchestra.run.smithi132.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-27T06:11:41.399 INFO:teuthology.orchestra.run.smithi132.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.9 MB/s | 144 kB 00:00 2024-03-27T06:11:41.441 INFO:teuthology.orchestra.run.smithi132.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 261 kB/s | 11 kB 00:00 2024-03-27T06:11:41.483 INFO:teuthology.orchestra.run.smithi132.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 432 kB/s | 18 kB 00:00 2024-03-27T06:11:41.550 INFO:teuthology.orchestra.run.smithi132.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.9 MB/s | 2.0 MB 00:00 2024-03-27T06:11:41.562 INFO:teuthology.orchestra.run.smithi134.stdout:(7/174): ceph-mon-19.0.0-2478.g155268c4.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-27T06:11:41.692 INFO:teuthology.orchestra.run.smithi132.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 111 kB/s | 23 kB 00:00 2024-03-27T06:11:41.736 INFO:teuthology.orchestra.run.smithi132.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 105 kB/s | 19 kB 00:00 2024-03-27T06:11:41.768 INFO:teuthology.orchestra.run.smithi132.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 264 kB/s | 20 kB 00:00 2024-03-27T06:11:41.810 INFO:teuthology.orchestra.run.smithi132.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-27T06:11:41.860 INFO:teuthology.orchestra.run.smithi132.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.3 MB/s | 1.0 MB 00:00 2024-03-27T06:11:41.885 INFO:teuthology.orchestra.run.smithi132.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-27T06:11:41.902 INFO:teuthology.orchestra.run.smithi132.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-03-27T06:11:41.936 INFO:teuthology.orchestra.run.smithi132.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.4 MB/s | 272 kB 00:00 2024-03-27T06:11:41.953 INFO:teuthology.orchestra.run.smithi132.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 332 kB/s | 16 kB 00:00 2024-03-27T06:11:41.986 INFO:teuthology.orchestra.run.smithi132.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-27T06:11:42.003 INFO:teuthology.orchestra.run.smithi132.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 567 kB/s | 28 kB 00:00 2024-03-27T06:11:42.020 INFO:teuthology.orchestra.run.smithi132.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 916 kB/s | 31 kB 00:00 2024-03-27T06:11:42.062 INFO:teuthology.orchestra.run.smithi132.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-27T06:11:42.079 INFO:teuthology.orchestra.run.smithi132.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-27T06:11:42.105 INFO:teuthology.orchestra.run.smithi132.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 869 kB/s | 36 kB 00:00 2024-03-27T06:11:42.155 INFO:teuthology.orchestra.run.smithi132.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-03-27T06:11:42.188 INFO:teuthology.orchestra.run.smithi132.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-27T06:11:42.219 INFO:teuthology.orchestra.run.smithi132.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.4 MB/s | 90 kB 00:00 2024-03-27T06:11:42.261 INFO:teuthology.orchestra.run.smithi132.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.8 MB/s | 427 kB 00:00 2024-03-27T06:11:42.287 INFO:teuthology.orchestra.run.smithi132.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 302 kB/s | 20 kB 00:00 2024-03-27T06:11:42.320 INFO:teuthology.orchestra.run.smithi132.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-27T06:11:42.396 INFO:teuthology.orchestra.run.smithi132.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-27T06:11:42.496 INFO:teuthology.orchestra.run.smithi134.stdout:(8/174): ceph-common-19.0.0-2478.g155268c4.el9. 9.0 MB/s | 21 MB 00:02 2024-03-27T06:11:42.613 INFO:teuthology.orchestra.run.smithi134.stdout:(9/174): ceph-radosgw-19.0.0-2478.g155268c4.el9 9.4 MB/s | 9.8 MB 00:01 2024-03-27T06:11:42.638 INFO:teuthology.orchestra.run.smithi134.stdout:(10/174): ceph-selinux-19.0.0-2478.g155268c4.el 177 kB/s | 25 kB 00:00 2024-03-27T06:11:42.758 INFO:teuthology.orchestra.run.smithi134.stdout:(11/174): cephfs-mirror-19.0.0-2478.g155268c4.e 1.8 MB/s | 221 kB 00:00 2024-03-27T06:11:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:(150/174): postgresql-server-13.14-1.el9.x86_64 1.2 MB/s | 5.8 MB 00:04 2024-03-27T06:11:42.864 INFO:teuthology.orchestra.run.smithi134.stdout:(12/174): cephfs-shell-19.0.0-2478.g155268c4.el 255 kB/s | 27 kB 00:00 2024-03-27T06:11:42.939 INFO:teuthology.orchestra.run.smithi132.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-03-27T06:11:42.947 INFO:teuthology.orchestra.run.smithi134.stdout:(13/174): libcephfs-devel-19.0.0-2478.g155268c4 382 kB/s | 32 kB 00:00 2024-03-27T06:11:43.022 INFO:teuthology.orchestra.run.smithi132.stdout:(152/174): librados2-19.0.0-2478.g155268c4.el9. 4.8 MB/s | 3.4 MB 00:00 2024-03-27T06:11:43.131 INFO:teuthology.orchestra.run.smithi134.stdout:(14/174): libcephfs2-19.0.0-2478.g155268c4.el9. 4.0 MB/s | 746 kB 00:00 2024-03-27T06:11:43.223 INFO:teuthology.orchestra.run.smithi134.stdout:(15/174): libcephsqlite-19.0.0-2478.g155268c4.e 1.7 MB/s | 163 kB 00:00 2024-03-27T06:11:43.264 INFO:teuthology.orchestra.run.smithi132.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.1 MB/s | 2.3 MB 00:00 2024-03-27T06:11:43.307 INFO:teuthology.orchestra.run.smithi134.stdout:(16/174): librados-devel-19.0.0-2478.g155268c4. 1.5 MB/s | 127 kB 00:00 2024-03-27T06:11:43.364 INFO:teuthology.orchestra.run.smithi132.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 5.4 MB/s | 1.8 MB 00:00 2024-03-27T06:11:43.381 INFO:teuthology.orchestra.run.smithi132.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 363 kB/s | 42 kB 00:00 2024-03-27T06:11:43.424 INFO:teuthology.orchestra.run.smithi134.stdout:(17/174): libradosstriper1-19.0.0-2478.g155268c 4.2 MB/s | 501 kB 00:00 2024-03-27T06:11:43.448 INFO:teuthology.orchestra.run.smithi132.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2024-03-27T06:11:43.481 INFO:teuthology.orchestra.run.smithi132.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-27T06:11:43.515 INFO:teuthology.orchestra.run.smithi132.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 552 kB/s | 36 kB 00:00 2024-03-27T06:11:43.574 INFO:teuthology.orchestra.run.smithi132.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-03-27T06:11:43.607 INFO:teuthology.orchestra.run.smithi132.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-27T06:11:43.650 INFO:teuthology.orchestra.run.smithi132.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 308 kB/s | 23 kB 00:00 2024-03-27T06:11:43.691 INFO:teuthology.orchestra.run.smithi132.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-27T06:11:43.783 INFO:teuthology.orchestra.run.smithi132.stdout:(163/174): librbd1-19.0.0-2478.g155268c4.el9.x8 2.3 MB/s | 3.1 MB 00:01 2024-03-27T06:11:43.825 INFO:teuthology.orchestra.run.smithi132.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 886 kB/s | 155 kB 00:00 2024-03-27T06:11:43.850 INFO:teuthology.orchestra.run.smithi132.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 857 kB/s | 136 kB 00:00 2024-03-27T06:11:43.875 INFO:teuthology.orchestra.run.smithi132.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 700 kB/s | 64 kB 00:00 2024-03-27T06:11:43.901 INFO:teuthology.orchestra.run.smithi132.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 384 kB/s | 29 kB 00:00 2024-03-27T06:11:44.074 INFO:teuthology.orchestra.run.smithi134.stdout:(18/174): librgw2-19.0.0-2478.g155268c4.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-03-27T06:11:44.076 INFO:teuthology.orchestra.run.smithi132.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 1.2 MB 00:00 2024-03-27T06:11:44.243 INFO:teuthology.orchestra.run.smithi132.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.9 MB/s | 2.2 MB 00:00 2024-03-27T06:11:44.266 INFO:teuthology.orchestra.run.smithi134.stdout:(19/174): ceph-osd-19.0.0-2478.g155268c4.el9.x8 5.5 MB/s | 17 MB 00:03 2024-03-27T06:11:44.291 INFO:teuthology.orchestra.run.smithi134.stdout:(20/174): python3-ceph-argparse-19.0.0-2478.g15 208 kB/s | 45 kB 00:00 2024-03-27T06:11:44.316 INFO:teuthology.orchestra.run.smithi134.stdout:(21/174): python3-ceph-common-19.0.0-2478.g1552 2.6 MB/s | 134 kB 00:00 2024-03-27T06:11:44.358 INFO:teuthology.orchestra.run.smithi134.stdout:(22/174): python3-cephfs-19.0.0-2478.g155268c4. 2.4 MB/s | 162 kB 00:00 2024-03-27T06:11:44.360 INFO:teuthology.orchestra.run.smithi132.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.0 MB/s | 2.3 MB 00:00 2024-03-27T06:11:44.385 INFO:teuthology.orchestra.run.smithi132.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 220 kB/s | 31 kB 00:00 2024-03-27T06:11:44.409 INFO:teuthology.orchestra.run.smithi134.stdout:(23/174): python3-rados-19.0.0-2478.g155268c4.e 3.4 MB/s | 320 kB 00:00 2024-03-27T06:11:44.419 INFO:teuthology.orchestra.run.smithi132.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 1.3 MB/s | 465 kB 00:00 2024-03-27T06:11:44.442 INFO:teuthology.orchestra.run.smithi134.stdout:(24/174): python3-rbd-19.0.0-2478.g155268c4.el9 3.5 MB/s | 298 kB 00:00 2024-03-27T06:11:44.468 INFO:teuthology.orchestra.run.smithi134.stdout:(25/174): python3-rgw-19.0.0-2478.g155268c4.el9 1.7 MB/s | 99 kB 00:00 2024-03-27T06:11:44.518 INFO:teuthology.orchestra.run.smithi134.stdout:(26/174): rbd-fuse-19.0.0-2478.g155268c4.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-27T06:11:44.618 INFO:teuthology.orchestra.run.smithi134.stdout:(27/174): rbd-nbd-19.0.0-2478.g155268c4.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-27T06:11:44.686 INFO:teuthology.orchestra.run.smithi132.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.3 MB/s | 1.1 MB 00:00 2024-03-27T06:11:44.710 INFO:teuthology.orchestra.run.smithi134.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2478.g 369 kB/s | 34 kB 00:00 2024-03-27T06:11:44.811 INFO:teuthology.orchestra.run.smithi134.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2478.g155268c 1.4 MB/s | 141 kB 00:00 2024-03-27T06:11:44.827 INFO:teuthology.orchestra.run.smithi132.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 168 kB/s | 74 kB 00:00 2024-03-27T06:11:44.839 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-27T06:11:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Total 12 MB/s | 227 MB 00:18 2024-03-27T06:11:44.978 INFO:teuthology.orchestra.run.smithi134.stdout:(30/174): rbd-mirror-19.0.0-2478.g155268c4.el9. 6.1 MB/s | 3.1 MB 00:00 2024-03-27T06:11:45.270 INFO:teuthology.orchestra.run.smithi134.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2478.g15526 7.7 MB/s | 3.5 MB 00:00 2024-03-27T06:11:45.353 INFO:teuthology.orchestra.run.smithi134.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2478.g15 2.9 MB/s | 249 kB 00:00 2024-03-27T06:11:45.437 INFO:teuthology.orchestra.run.smithi134.stdout:(33/174): ceph-mgr-rook-19.0.0-2478.g155268c4.e 591 kB/s | 49 kB 00:00 2024-03-27T06:11:45.520 INFO:teuthology.orchestra.run.smithi134.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2478.g1 198 kB/s | 16 kB 00:00 2024-03-27T06:11:45.971 INFO:teuthology.orchestra.run.smithi134.stdout:(35/174): ceph-test-19.0.0-2478.g155268c4.el9.x 14 MB/s | 48 MB 00:03 2024-03-27T06:11:45.996 INFO:teuthology.orchestra.run.smithi134.stdout:(36/174): ceph-volume-19.0.0-2478.g155268c4.el9 555 kB/s | 264 kB 00:00 2024-03-27T06:11:46.055 INFO:teuthology.orchestra.run.smithi134.stdout:(37/174): cephfs-top-19.0.0-2478.g155268c4.el9. 313 kB/s | 18 kB 00:00 2024-03-27T06:11:46.089 INFO:teuthology.orchestra.run.smithi134.stdout:(38/174): cephadm-19.0.0-2478.g155268c4.el9.noa 6.4 MB/s | 762 kB 00:00 2024-03-27T06:11:46.230 INFO:teuthology.orchestra.run.smithi134.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-27T06:11:46.273 INFO:teuthology.orchestra.run.smithi134.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.7 MB/s | 920 kB 00:00 2024-03-27T06:11:46.298 INFO:teuthology.orchestra.run.smithi134.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-27T06:11:46.323 INFO:teuthology.orchestra.run.smithi134.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-27T06:11:46.350 INFO:teuthology.orchestra.run.smithi134.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2024-03-27T06:11:46.460 INFO:teuthology.orchestra.run.smithi134.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 5.0 MB/s | 7.4 MB 00:01 2024-03-27T06:11:46.489 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-27T06:11:46.491 INFO:teuthology.orchestra.run.smithi134.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 122 kB/s | 17 kB 00:00 2024-03-27T06:11:46.549 INFO:teuthology.orchestra.run.smithi134.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-03-27T06:11:46.575 INFO:teuthology.orchestra.run.smithi134.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 10 kB 00:00 2024-03-27T06:11:46.596 INFO:teuthology.orchestra.run.smithi134.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2024-03-27T06:11:46.622 INFO:teuthology.orchestra.run.smithi134.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 951 kB/s | 44 kB 00:00 2024-03-27T06:11:46.655 INFO:teuthology.orchestra.run.smithi134.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.1 MB/s | 661 kB 00:00 2024-03-27T06:11:46.681 INFO:teuthology.orchestra.run.smithi134.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-27T06:11:46.706 INFO:teuthology.orchestra.run.smithi134.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 563 kB/s | 46 kB 00:00 2024-03-27T06:11:46.739 INFO:teuthology.orchestra.run.smithi134.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-27T06:11:46.773 INFO:teuthology.orchestra.run.smithi134.stdout:(54/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-27T06:11:46.798 INFO:teuthology.orchestra.run.smithi134.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-03-27T06:11:46.823 INFO:teuthology.orchestra.run.smithi134.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 412 kB/s | 34 kB 00:00 2024-03-27T06:11:46.871 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-27T06:11:46.872 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-27T06:11:46.991 INFO:teuthology.orchestra.run.smithi134.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-27T06:11:47.024 INFO:teuthology.orchestra.run.smithi134.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-27T06:11:47.049 INFO:teuthology.orchestra.run.smithi134.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 347 kB/s | 78 kB 00:00 2024-03-27T06:11:47.191 INFO:teuthology.orchestra.run.smithi134.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-27T06:11:47.216 INFO:teuthology.orchestra.run.smithi134.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 149 kB/s | 24 kB 00:00 2024-03-27T06:11:47.242 INFO:teuthology.orchestra.run.smithi134.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 95 kB/s | 18 kB 00:00 2024-03-27T06:11:47.267 INFO:teuthology.orchestra.run.smithi134.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-27T06:11:47.292 INFO:teuthology.orchestra.run.smithi134.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-27T06:11:47.384 INFO:teuthology.orchestra.run.smithi134.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-27T06:11:47.409 INFO:teuthology.orchestra.run.smithi134.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-27T06:11:47.434 INFO:teuthology.orchestra.run.smithi134.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-27T06:11:47.460 INFO:teuthology.orchestra.run.smithi134.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-03-27T06:11:47.485 INFO:teuthology.orchestra.run.smithi134.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-27T06:11:47.510 INFO:teuthology.orchestra.run.smithi134.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-27T06:11:47.535 INFO:teuthology.orchestra.run.smithi134.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-27T06:11:47.561 INFO:teuthology.orchestra.run.smithi134.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-27T06:11:47.586 INFO:teuthology.orchestra.run.smithi134.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-27T06:11:47.611 INFO:teuthology.orchestra.run.smithi134.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 566 kB/s | 43 kB 00:00 2024-03-27T06:11:47.670 INFO:teuthology.orchestra.run.smithi134.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 465 kB/s | 39 kB 00:00 2024-03-27T06:11:47.753 INFO:teuthology.orchestra.run.smithi134.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 295 kB/s | 41 kB 00:00 2024-03-27T06:11:47.920 INFO:teuthology.orchestra.run.smithi134.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-27T06:11:47.945 INFO:teuthology.orchestra.run.smithi134.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-27T06:11:47.962 INFO:teuthology.orchestra.run.smithi134.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 273 kB/s | 56 kB 00:00 2024-03-27T06:11:47.979 INFO:teuthology.orchestra.run.smithi134.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-27T06:11:48.005 INFO:teuthology.orchestra.run.smithi134.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-27T06:11:48.021 INFO:teuthology.orchestra.run.smithi134.stdout:(82/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 752 kB/s | 44 kB 00:00 2024-03-27T06:11:48.047 INFO:teuthology.orchestra.run.smithi134.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-27T06:11:48.089 INFO:teuthology.orchestra.run.smithi134.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-27T06:11:48.114 INFO:teuthology.orchestra.run.smithi134.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2024-03-27T06:11:48.139 INFO:teuthology.orchestra.run.smithi134.stdout:(86/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-27T06:11:48.164 INFO:teuthology.orchestra.run.smithi134.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-27T06:11:48.298 INFO:teuthology.orchestra.run.smithi134.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-27T06:11:48.323 INFO:teuthology.orchestra.run.smithi134.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-27T06:11:48.373 INFO:teuthology.orchestra.run.smithi134.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-27T06:11:48.399 INFO:teuthology.orchestra.run.smithi134.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-03-27T06:11:48.441 INFO:teuthology.orchestra.run.smithi134.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-03-27T06:11:48.474 INFO:teuthology.orchestra.run.smithi134.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-27T06:11:48.658 INFO:teuthology.orchestra.run.smithi134.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-03-27T06:11:48.733 INFO:teuthology.orchestra.run.smithi134.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-03-27T06:11:48.767 INFO:teuthology.orchestra.run.smithi134.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-03-27T06:11:48.859 INFO:teuthology.orchestra.run.smithi134.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-27T06:11:48.917 INFO:teuthology.orchestra.run.smithi134.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.0 MB/s | 4.1 MB 00:00 2024-03-27T06:11:48.993 INFO:teuthology.orchestra.run.smithi134.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-03-27T06:11:49.026 INFO:teuthology.orchestra.run.smithi134.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-03-27T06:11:49.068 INFO:teuthology.orchestra.run.smithi134.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 478 kB/s | 36 kB 00:00 2024-03-27T06:11:49.093 INFO:teuthology.orchestra.run.smithi134.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2024-03-27T06:11:49.152 INFO:teuthology.orchestra.run.smithi134.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-03-27T06:11:49.267 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-27T06:11:49.268 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-27T06:11:49.285 INFO:teuthology.orchestra.run.smithi134.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-03-27T06:11:49.394 INFO:teuthology.orchestra.run.smithi134.stdout:(105/174): postgresql-server-13.14-1.el9.x86_64 8.7 MB/s | 5.8 MB 00:00 2024-03-27T06:11:49.445 INFO:teuthology.orchestra.run.smithi134.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.9 MB/s | 863 kB 00:00 2024-03-27T06:11:49.611 INFO:teuthology.orchestra.run.smithi134.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-03-27T06:11:49.736 INFO:teuthology.orchestra.run.smithi134.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-27T06:11:49.762 INFO:teuthology.orchestra.run.smithi134.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 841 kB/s | 309 kB 00:00 2024-03-27T06:11:49.804 INFO:teuthology.orchestra.run.smithi134.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-03-27T06:11:49.837 INFO:teuthology.orchestra.run.smithi134.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-27T06:11:49.879 INFO:teuthology.orchestra.run.smithi134.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 898 kB/s | 67 kB 00:00 2024-03-27T06:11:49.921 INFO:teuthology.orchestra.run.smithi134.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-27T06:11:50.022 INFO:teuthology.orchestra.run.smithi134.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-27T06:11:50.055 INFO:teuthology.orchestra.run.smithi134.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-27T06:11:50.122 INFO:teuthology.orchestra.run.smithi134.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-03-27T06:11:50.139 INFO:teuthology.orchestra.run.smithi134.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-03-27T06:11:50.181 INFO:teuthology.orchestra.run.smithi134.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-03-27T06:11:50.198 INFO:teuthology.orchestra.run.smithi134.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-27T06:11:50.257 INFO:teuthology.orchestra.run.smithi134.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-27T06:11:50.282 INFO:teuthology.orchestra.run.smithi134.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-27T06:11:50.307 INFO:teuthology.orchestra.run.smithi134.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-27T06:11:50.332 INFO:teuthology.orchestra.run.smithi134.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-03-27T06:11:50.357 INFO:teuthology.orchestra.run.smithi134.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-27T06:11:50.416 INFO:teuthology.orchestra.run.smithi134.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-27T06:11:50.441 INFO:teuthology.orchestra.run.smithi134.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-27T06:11:50.467 INFO:teuthology.orchestra.run.smithi134.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-27T06:11:50.492 INFO:teuthology.orchestra.run.smithi134.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-27T06:11:50.517 INFO:teuthology.orchestra.run.smithi134.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-03-27T06:11:50.543 INFO:teuthology.orchestra.run.smithi134.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-03-27T06:11:50.568 INFO:teuthology.orchestra.run.smithi134.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-27T06:11:50.601 INFO:teuthology.orchestra.run.smithi134.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-27T06:11:50.626 INFO:teuthology.orchestra.run.smithi134.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-03-27T06:11:50.654 INFO:teuthology.orchestra.run.smithi134.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 940 kB/s | 79 kB 00:00 2024-03-27T06:11:50.677 INFO:teuthology.orchestra.run.smithi134.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-03-27T06:11:50.702 INFO:teuthology.orchestra.run.smithi134.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-27T06:11:50.736 INFO:teuthology.orchestra.run.smithi134.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 195 kB/s | 16 kB 00:00 2024-03-27T06:11:50.762 INFO:teuthology.orchestra.run.smithi134.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-27T06:11:50.787 INFO:teuthology.orchestra.run.smithi134.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 335 kB/s | 28 kB 00:00 2024-03-27T06:11:50.812 INFO:teuthology.orchestra.run.smithi134.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-27T06:11:50.837 INFO:teuthology.orchestra.run.smithi134.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-27T06:11:50.854 INFO:teuthology.orchestra.run.smithi134.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-27T06:11:50.879 INFO:teuthology.orchestra.run.smithi134.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 534 kB/s | 36 kB 00:00 2024-03-27T06:11:50.896 INFO:teuthology.orchestra.run.smithi134.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-03-27T06:11:50.929 INFO:teuthology.orchestra.run.smithi134.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-27T06:11:50.955 INFO:teuthology.orchestra.run.smithi134.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-27T06:11:50.980 INFO:teuthology.orchestra.run.smithi134.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-27T06:11:51.005 INFO:teuthology.orchestra.run.smithi134.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-27T06:11:51.030 INFO:teuthology.orchestra.run.smithi134.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-27T06:11:51.073 INFO:teuthology.orchestra.run.smithi134.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-27T06:11:51.140 INFO:teuthology.orchestra.run.smithi134.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.8 MB/s | 321 kB 00:00 2024-03-27T06:11:51.390 INFO:teuthology.orchestra.run.smithi134.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.3 MB/s | 2.3 MB 00:00 2024-03-27T06:11:51.465 INFO:teuthology.orchestra.run.smithi134.stdout:(153/174): librados2-19.0.0-2478.g155268c4.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-27T06:11:51.549 INFO:teuthology.orchestra.run.smithi134.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 506 kB/s | 42 kB 00:00 2024-03-27T06:11:51.599 INFO:teuthology.orchestra.run.smithi134.stdout:(155/174): librbd1-19.0.0-2478.g155268c4.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-03-27T06:11:51.641 INFO:teuthology.orchestra.run.smithi134.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 7.3 MB/s | 1.8 MB 00:00 2024-03-27T06:11:51.666 INFO:teuthology.orchestra.run.smithi134.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-27T06:11:51.692 INFO:teuthology.orchestra.run.smithi134.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 775 kB/s | 71 kB 00:00 2024-03-27T06:11:51.717 INFO:teuthology.orchestra.run.smithi134.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 483 kB/s | 36 kB 00:00 2024-03-27T06:11:51.742 INFO:teuthology.orchestra.run.smithi134.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-27T06:11:51.767 INFO:teuthology.orchestra.run.smithi134.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-27T06:11:51.792 INFO:teuthology.orchestra.run.smithi134.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-27T06:11:51.818 INFO:teuthology.orchestra.run.smithi134.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-27T06:11:51.843 INFO:teuthology.orchestra.run.smithi134.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-27T06:11:51.868 INFO:teuthology.orchestra.run.smithi134.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-27T06:11:51.894 INFO:teuthology.orchestra.run.smithi134.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 846 kB/s | 64 kB 00:00 2024-03-27T06:11:51.919 INFO:teuthology.orchestra.run.smithi134.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-27T06:11:52.078 INFO:teuthology.orchestra.run.smithi134.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 1.2 MB 00:00 2024-03-27T06:11:52.195 INFO:teuthology.orchestra.run.smithi134.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2024-03-27T06:11:52.220 INFO:teuthology.orchestra.run.smithi134.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-27T06:11:52.245 INFO:teuthology.orchestra.run.smithi134.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 623 kB/s | 31 kB 00:00 2024-03-27T06:11:52.287 INFO:teuthology.orchestra.run.smithi134.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.2 MB/s | 2.3 MB 00:00 2024-03-27T06:11:52.346 INFO:teuthology.orchestra.run.smithi134.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-27T06:11:52.371 INFO:teuthology.orchestra.run.smithi134.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-27T06:11:52.376 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-27T06:11:52.376 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 227 MB 00:12 2024-03-27T06:11:52.940 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-27T06:11:53.217 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-27T06:11:53.217 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-27T06:11:54.961 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-27T06:11:54.961 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-27T06:11:56.736 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-27T06:11:56.876 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-27T06:11:56.924 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-27T06:11:56.962 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-27T06:11:57.006 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-27T06:11:57.092 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-27T06:11:57.234 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-27T06:11:57.288 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-27T06:11:57.320 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-27T06:11:57.627 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-27T06:11:58.050 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-27T06:11:58.103 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:11:58.252 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:11:58.287 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:11:58.435 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:11:58.477 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 12/198 2024-03-27T06:11:58.540 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-27T06:11:58.834 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-27T06:11:58.893 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-27T06:11:58.919 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:11:59.085 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:11:59.112 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 17/198 2024-03-27T06:11:59.227 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 17/198 2024-03-27T06:11:59.281 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-27T06:11:59.314 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-27T06:11:59.374 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-27T06:11:59.690 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-27T06:11:59.739 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-27T06:11:59.789 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-27T06:12:00.096 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-27T06:12:00.280 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-27T06:12:00.384 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-27T06:12:00.553 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:00.599 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:00.927 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:01.120 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-27T06:12:01.253 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-27T06:12:01.289 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-27T06:12:01.325 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-27T06:12:01.390 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-27T06:12:01.510 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-27T06:12:01.596 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-27T06:12:01.640 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-27T06:12:01.739 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-27T06:12:01.786 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-27T06:12:01.849 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-27T06:12:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-27T06:12:01.954 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-27T06:12:01.973 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-27T06:12:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-27T06:12:02.087 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-27T06:12:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-27T06:12:02.127 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-2478.g155268c4.el9. 39/198 2024-03-27T06:12:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-27T06:12:02.165 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-27T06:12:02.176 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-2478.g155268c4.el 40/198 2024-03-27T06:12:02.206 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-27T06:12:02.208 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_6 41/198 2024-03-27T06:12:02.259 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-27T06:12:02.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-27T06:12:02.321 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-27T06:12:02.326 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-27T06:12:02.368 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-27T06:12:02.387 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-27T06:12:02.486 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-27T06:12:02.655 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-27T06:12:02.688 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-27T06:12:02.752 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:12:02.755 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-27T06:12:02.873 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:12:02.912 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:12:02.986 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:12:03.024 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 12/198 2024-03-27T06:12:03.065 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-27T06:12:03.240 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-27T06:12:03.305 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-27T06:12:03.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:12:03.429 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:12:03.449 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-27T06:12:03.451 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 17/198 2024-03-27T06:12:03.498 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 17/198 2024-03-27T06:12:03.530 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-27T06:12:03.552 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-27T06:12:03.564 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-27T06:12:03.568 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-27T06:12:03.696 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-27T06:12:03.718 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-27T06:12:03.750 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-27T06:12:03.755 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-27T06:12:03.785 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-27T06:12:03.790 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-27T06:12:03.837 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-27T06:12:03.895 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-27T06:12:03.907 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-27T06:12:03.965 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-2478.g155268c4.el9.x86_6 53/198 2024-03-27T06:12:03.975 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-27T06:12:04.013 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-27T06:12:04.027 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-27T06:12:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:04.162 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:04.361 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-27T06:12:04.450 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-27T06:12:04.475 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-27T06:12:04.569 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-27T06:12:04.599 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-27T06:12:04.627 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-27T06:12:04.661 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-27T06:12:04.662 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-27T06:12:04.734 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-27T06:12:04.789 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-27T06:12:04.818 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-27T06:12:04.828 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-27T06:12:04.876 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-27T06:12:04.890 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-27T06:12:04.927 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-27T06:12:04.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-27T06:12:05.028 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-27T06:12:05.056 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-27T06:12:05.097 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-27T06:12:05.125 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-27T06:12:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-2478.g155268c4.el9. 39/198 2024-03-27T06:12:05.182 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-27T06:12:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-2478.g155268c4.el 40/198 2024-03-27T06:12:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_6 41/198 2024-03-27T06:12:05.262 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-27T06:12:05.318 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-27T06:12:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-27T06:12:05.372 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-27T06:12:05.390 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-27T06:12:05.429 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-27T06:12:05.445 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-27T06:12:05.529 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-27T06:12:05.578 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-27T06:12:05.581 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-27T06:12:05.624 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-27T06:12:05.665 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-27T06:12:05.736 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-27T06:12:05.937 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-27T06:12:05.994 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-27T06:12:06.033 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-27T06:12:06.091 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-27T06:12:06.139 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-27T06:12:06.221 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-27T06:12:06.231 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-27T06:12:06.298 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-27T06:12:06.314 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-27T06:12:06.349 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-27T06:12:06.352 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-27T06:12:06.358 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-27T06:12:06.388 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-27T06:12:06.389 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-27T06:12:06.422 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-27T06:12:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-27T06:12:06.473 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-2478.g155268c4.el9.x86_6 53/198 2024-03-27T06:12:06.503 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-27T06:12:06.593 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-27T06:12:06.741 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-27T06:12:06.773 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-27T06:12:06.894 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-27T06:12:06.903 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-27T06:12:06.955 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-27T06:12:06.957 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-27T06:12:06.998 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-27T06:12:07.028 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-27T06:12:07.124 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-27T06:12:07.147 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-27T06:12:07.209 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-27T06:12:07.263 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-27T06:12:07.328 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-27T06:12:07.360 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-27T06:12:07.377 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-27T06:12:07.446 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-27T06:12:07.641 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-27T06:12:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-27T06:12:07.700 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-27T06:12:07.742 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-27T06:12:07.772 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-27T06:12:07.783 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-27T06:12:07.814 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-27T06:12:07.825 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-27T06:12:07.857 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-27T06:12:07.896 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-27T06:12:07.900 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-27T06:12:07.966 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-27T06:12:08.023 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-27T06:12:08.035 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-27T06:12:08.070 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-27T06:12:08.086 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-27T06:12:08.117 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-27T06:12:08.158 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-27T06:12:08.199 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-27T06:12:08.240 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-27T06:12:08.250 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-27T06:12:08.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-27T06:12:08.301 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-27T06:12:08.332 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-27T06:12:08.366 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-27T06:12:08.374 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-27T06:12:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-27T06:12:08.413 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-27T06:12:08.470 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-27T06:12:08.559 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-27T06:12:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-27T06:12:08.671 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-27T06:12:08.697 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-27T06:12:08.733 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-27T06:12:08.757 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-27T06:12:08.770 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-27T06:12:08.805 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-27T06:12:08.825 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-27T06:12:08.860 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-27T06:12:08.975 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-27T06:12:09.043 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-27T06:12:09.051 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-27T06:12:09.103 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-27T06:12:09.228 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-27T06:12:09.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-27T06:12:09.307 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-27T06:12:09.347 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-27T06:12:09.374 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-27T06:12:09.409 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-27T06:12:09.448 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-27T06:12:09.479 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-27T06:12:09.510 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-27T06:12:09.585 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-27T06:12:09.620 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-27T06:12:09.653 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-27T06:12:09.694 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-27T06:12:09.729 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-27T06:12:09.825 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-27T06:12:09.915 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-27T06:12:09.959 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-27T06:12:09.991 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-27T06:12:10.000 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-27T06:12:10.038 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-27T06:12:10.051 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-27T06:12:10.084 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-27T06:12:10.134 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-27T06:12:10.148 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-27T06:12:10.193 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-27T06:12:10.270 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-27T06:12:10.329 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-27T06:12:10.565 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-27T06:12:10.621 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-27T06:12:10.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-27T06:12:10.913 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-27T06:12:10.957 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-27T06:12:10.990 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-27T06:12:11.039 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-27T06:12:11.126 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-27T06:12:11.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-27T06:12:11.357 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-27T06:12:11.389 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-27T06:12:11.467 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-27T06:12:11.569 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-27T06:12:11.703 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-27T06:12:12.102 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-27T06:12:12.220 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-27T06:12:13.218 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-27T06:12:13.313 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-27T06:12:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-27T06:12:13.407 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-27T06:12:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-27T06:12:13.456 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-27T06:12:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-27T06:12:13.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-27T06:12:13.584 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-27T06:12:13.611 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-27T06:12:13.846 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-27T06:12:13.887 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 124/198 2024-03-27T06:12:13.921 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-27T06:12:13.950 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 124/198 2024-03-27T06:12:14.007 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 124/198 2024-03-27T06:12:14.040 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 124/198 2024-03-27T06:12:14.044 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 125/198 2024-03-27T06:12:14.071 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 125/198 2024-03-27T06:12:14.099 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-27T06:12:14.107 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-27T06:12:14.567 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-27T06:12:14.594 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 128/198 2024-03-27T06:12:14.627 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-27T06:12:14.648 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 128/198 2024-03-27T06:12:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 129/198 2024-03-27T06:12:14.866 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 128/198 2024-03-27T06:12:14.969 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 128/198 2024-03-27T06:12:15.002 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 129/198 2024-03-27T06:12:16.948 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:17.000 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:17.535 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:17.570 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 131/198 2024-03-27T06:12:17.652 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 131/198 2024-03-27T06:12:17.652 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T06:12:17.652 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T06:12:17.652 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:17.666 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:17.680 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 132/198 2024-03-27T06:12:17.714 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 130/198 2024-03-27T06:12:18.393 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 131/198 2024-03-27T06:12:18.557 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 131/198 2024-03-27T06:12:18.557 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T06:12:18.557 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T06:12:18.557 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:18.610 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 132/198 2024-03-27T06:12:29.492 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 132/198 2024-03-27T06:12:29.492 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-03-27T06:12:29.492 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-03-27T06:12:29.492 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-03-27T06:12:29.493 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-03-27T06:12:29.493 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-03-27T06:12:29.493 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-03-27T06:12:29.493 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-03-27T06:12:29.493 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:29.659 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 133/198 2024-03-27T06:12:29.677 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 133/198 2024-03-27T06:12:29.677 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:29.677 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T06:12:29.677 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:31.137 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 134/198 2024-03-27T06:12:31.162 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 134/198 2024-03-27T06:12:31.162 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:31.162 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T06:12:31.163 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T06:12:31.163 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T06:12:31.163 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:31.387 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 135/198 2024-03-27T06:12:31.409 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 135/198 2024-03-27T06:12:31.409 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:31.409 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T06:12:31.409 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T06:12:31.410 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T06:12:31.410 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:31.526 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 132/198 2024-03-27T06:12:31.526 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-03-27T06:12:31.526 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-03-27T06:12:31.526 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-03-27T06:12:31.526 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-03-27T06:12:31.527 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-03-27T06:12:31.527 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-03-27T06:12:31.527 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-03-27T06:12:31.527 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:31.774 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 133/198 2024-03-27T06:12:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 133/198 2024-03-27T06:12:31.823 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:31.823 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T06:12:31.823 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:31.855 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 136/198 2024-03-27T06:12:31.877 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 136/198 2024-03-27T06:12:31.877 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:31.877 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T06:12:31.877 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T06:12:31.878 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T06:12:31.878 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-27T06:12:32.384 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-2478.g155268c4.el9.noarch 138/198 2024-03-27T06:12:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-2478.g155268c4.el9.noarch 138/198 2024-03-27T06:12:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.e 139/198 2024-03-27T06:12:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2478.g155268c4. 140/198 2024-03-27T06:12:32.536 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 141/198 2024-03-27T06:12:33.479 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 134/198 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 134/198 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T06:12:33.552 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:33.572 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 141/198 2024-03-27T06:12:33.613 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 142/198 2024-03-27T06:12:33.821 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 135/198 2024-03-27T06:12:33.890 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 135/198 2024-03-27T06:12:33.891 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:33.891 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T06:12:33.891 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T06:12:33.891 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T06:12:33.891 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:34.380 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 136/198 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 136/198 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T06:12:34.455 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:34.573 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-27T06:12:34.578 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 142/198 2024-03-27T06:12:34.622 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 143/198 2024-03-27T06:12:34.722 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 143/198 2024-03-27T06:12:34.893 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el 144/198 2024-03-27T06:12:34.932 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 145/198 2024-03-27T06:12:34.955 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 145/198 2024-03-27T06:12:34.955 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:34.956 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T06:12:34.956 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T06:12:34.956 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T06:12:34.956 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:34.999 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 146/198 2024-03-27T06:12:35.010 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 146/198 2024-03-27T06:12:35.055 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-2478.g155268c4.el9.x86_64 147/198 2024-03-27T06:12:35.213 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-2478.g155268c4.el9.noarch 138/198 2024-03-27T06:12:35.249 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-2478.g155268c4.el9.noarch 138/198 2024-03-27T06:12:35.308 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.e 139/198 2024-03-27T06:12:35.429 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2478.g155268c4. 140/198 2024-03-27T06:12:35.470 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 141/198 2024-03-27T06:12:35.923 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-27T06:12:35.951 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 149/198 2024-03-27T06:12:35.973 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 149/198 2024-03-27T06:12:35.973 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:35.974 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T06:12:35.974 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T06:12:35.974 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T06:12:35.974 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:36.019 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2478.g15526 150/198 2024-03-27T06:12:36.040 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2478.g15526 150/198 2024-03-27T06:12:36.040 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:36.040 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T06:12:36.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:36.094 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-27T06:12:36.116 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:36.420 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 152/198 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 152/198 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T06:12:36.442 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:36.729 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 141/198 2024-03-27T06:12:36.772 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 142/198 2024-03-27T06:12:37.850 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 142/198 2024-03-27T06:12:37.881 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 143/198 2024-03-27T06:12:38.019 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 143/198 2024-03-27T06:12:38.194 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el 144/198 2024-03-27T06:12:38.233 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 145/198 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 145/198 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T06:12:38.302 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:38.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 146/198 2024-03-27T06:12:38.403 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 146/198 2024-03-27T06:12:38.466 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-2478.g155268c4.el9.x86_64 147/198 2024-03-27T06:12:39.464 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-27T06:12:39.504 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 149/198 2024-03-27T06:12:39.572 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 149/198 2024-03-27T06:12:39.572 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:39.572 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T06:12:39.572 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T06:12:39.573 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T06:12:39.573 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:39.630 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2478.g15526 150/198 2024-03-27T06:12:39.698 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2478.g15526 150/198 2024-03-27T06:12:39.698 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:39.698 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T06:12:39.698 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:39.773 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-27T06:12:39.847 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:40.240 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 152/198 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 152/198 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T06:12:40.307 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:40.865 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 153/198 2024-03-27T06:12:40.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 154/198 2024-03-27T06:12:40.945 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 155/198 2024-03-27T06:12:40.974 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-27T06:12:41.150 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 157/198 2024-03-27T06:12:41.263 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-27T06:12:41.303 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 159/198 2024-03-27T06:12:41.342 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-27T06:12:41.395 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-27T06:12:41.438 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-27T06:12:41.463 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_ 163/198 2024-03-27T06:12:41.495 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-27T06:12:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-27T06:12:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-27T06:12:41.639 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-27T06:12:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-27T06:12:42.276 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:42.341 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:42.361 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:43.175 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 170/198 2024-03-27T06:12:43.265 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-27T06:12:43.305 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-27T06:12:43.387 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-27T06:12:43.427 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-27T06:12:43.460 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-27T06:12:43.486 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-27T06:12:43.512 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-27T06:12:43.512 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-27T06:12:43.600 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-27T06:12:43.629 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-27T06:12:43.630 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-27T06:12:43.714 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-27T06:12:43.741 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-27T06:12:43.782 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-27T06:12:43.808 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-27T06:12:43.833 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-27T06:12:43.865 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-27T06:12:43.905 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-27T06:12:43.937 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-27T06:12:43.970 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-27T06:12:43.996 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-27T06:12:44.021 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-27T06:12:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-27T06:12:44.052 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:44.052 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:44.249 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:44.249 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-27T06:12:44.332 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-27T06:12:44.358 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-27T06:12:44.389 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-27T06:12:44.425 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-27T06:12:44.458 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-27T06:12:44.522 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-27T06:12:45.113 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 153/198 2024-03-27T06:12:45.157 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 154/198 2024-03-27T06:12:45.210 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 155/198 2024-03-27T06:12:45.248 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-27T06:12:45.515 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 157/198 2024-03-27T06:12:45.666 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-27T06:12:45.720 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 159/198 2024-03-27T06:12:45.769 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-27T06:12:45.845 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-27T06:12:45.890 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-27T06:12:45.925 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_ 163/198 2024-03-27T06:12:45.995 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-27T06:12:46.047 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-27T06:12:46.143 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-27T06:12:46.264 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-27T06:12:46.305 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-27T06:12:47.640 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:47.694 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:47.751 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-27T06:12:49.029 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 170/198 2024-03-27T06:12:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-27T06:12:49.217 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-27T06:12:49.378 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-27T06:12:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-27T06:12:49.460 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-27T06:12:49.494 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-27T06:12:49.536 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-27T06:12:49.536 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-27T06:12:49.689 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-27T06:12:49.732 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-27T06:12:49.732 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-27T06:12:49.861 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-27T06:12:49.906 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-2478.g155268c4.el9.x86_64 1/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 2/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 3/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 4/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2478.g15526 5/198 2024-03-27T06:12:49.949 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 6/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 7/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 8/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 9/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 12/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 13/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 14/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_ 15/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 17/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-2478.g155268c4.el9.x86_6 18/198 2024-03-27T06:12:49.950 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 19/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 20/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2478.g155268c4.el 21/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-2478.g155268c4.el9. 22/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_6 23/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 24/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 25/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 26/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 27/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 28/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 29/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2478.g155268c4. 30/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 31/198 2024-03-27T06:12:49.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 32/198 2024-03-27T06:12:49.966 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-27T06:12:49.967 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 33/198 2024-03-27T06:12:49.967 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el 34/198 2024-03-27T06:12:49.967 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 35/198 2024-03-27T06:12:49.967 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.e 36/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 37/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-2478.g155268c4.el9.noarch 38/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 39/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-27T06:12:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-27T06:12:49.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-27T06:12:49.970 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-27T06:12:49.971 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-27T06:12:49.972 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-27T06:12:49.973 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-27T06:12:49.974 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-27T06:12:49.975 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:49.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 153/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-27T06:12:49.977 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-27T06:12:49.978 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-27T06:12:49.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-27T06:12:49.980 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-27T06:12:50.026 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-27T06:12:50.059 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-27T06:12:50.129 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-27T06:12:50.197 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-27T06:12:50.239 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-27T06:12:50.280 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-27T06:12:50.330 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-27T06:12:50.385 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-27T06:12:50.423 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-27T06:12:50.459 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:50.460 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:50.812 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-27T06:12:50.812 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-27T06:12:50.954 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-27T06:12:50.985 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-27T06:12:51.039 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-27T06:12:51.070 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-27T06:12:51.119 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-27T06:12:51.239 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-27T06:12:51.499 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-27T06:12:51.499 INFO:teuthology.orchestra.run.smithi132.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-27T06:12:51.499 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: acl-2.3.1-4.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-27T06:12:53.087 INFO:teuthology.orchestra.run.smithi134.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.088 INFO:teuthology.orchestra.run.smithi134.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: automake-1.16.2-8.el9.noarch 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: bison-3.7.4-5.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.089 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.091 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: flex-2.6.4-9.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T06:12:53.092 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.093 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T06:12:53.094 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-13.14-1.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T06:12:53.095 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T06:12:53.096 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T06:12:53.097 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T06:12:53.098 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T06:12:53.099 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:53.100 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:12:53.101 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-27T06:12:53.588 DEBUG:teuthology.parallel:result is None 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-2478.g155268c4.el9.x86_64 1/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 2/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 3/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 4/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2478.g15526 5/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 6/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 7/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 8/198 2024-03-27T06:12:56.846 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 9/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 10/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 11/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 12/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 13/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 14/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_ 15/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 16/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 17/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-2478.g155268c4.el9.x86_6 18/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86 19/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 20/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2478.g155268c4.el 21/198 2024-03-27T06:12:56.847 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-2478.g155268c4.el9. 22/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_6 23/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 24/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 25/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 26/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 27/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 28/198 2024-03-27T06:12:56.848 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 29/198 2024-03-27T06:12:56.863 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2478.g155268c4. 30/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noa 31/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.n 32/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2478.g155 33/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el 34/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 35/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.e 36/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 37/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-2478.g155268c4.el9.noarch 38/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 39/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-27T06:12:56.864 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-27T06:12:56.865 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-27T06:12:56.866 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-27T06:12:56.867 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-27T06:12:56.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-27T06:12:56.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-27T06:12:56.870 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-27T06:12:56.871 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-2478.g155268c4.el9.x86_64 151/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 153/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-27T06:12:56.872 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-27T06:12:56.873 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-27T06:12:56.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-27T06:12:56.875 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-27T06:12:56.876 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-27T06:12:59.562 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: acl-2.3.1-4.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-27T06:12:59.563 INFO:teuthology.orchestra.run.smithi132.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-27T06:12:59.564 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: automake-1.16.2-8.el9.noarch 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: bison-3.7.4-5.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.565 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.566 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top-2:19.0.0-2478.g155268c4.el9.noarch 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: flex-2.6.4-9.el9.x86_64 2024-03-27T06:12:59.570 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-16.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.571 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.572 INFO:teuthology.orchestra.run.smithi132.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-27T06:12:59.573 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-13.14-1.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T06:12:59.574 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T06:12:59.575 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T06:12:59.576 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T06:12:59.577 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T06:12:59.578 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T06:12:59.579 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-2478.g155268c4.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-27T06:12:59.580 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T06:12:59.581 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-27T06:12:59.581 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:12:59.581 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-27T06:13:00.452 DEBUG:teuthology.parallel:result is None 2024-03-27T06:13:00.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:13:00.674 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T06:13:00.736 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-2478.g155268c4.el9 2024-03-27T06:13:00.737 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2478.g155268c4.el9 2024-03-27T06:13:00.737 INFO:teuthology.task.install:The correct ceph version 19.0.0-2478.g155268c4 is installed. 2024-03-27T06:13:00.739 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:13:00.879 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T06:13:00.904 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-2478.g155268c4.el9 2024-03-27T06:13:00.905 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2478.g155268c4.el9 2024-03-27T06:13:00.905 INFO:teuthology.task.install:The correct ceph version 19.0.0-2478.g155268c4 is installed. 2024-03-27T06:13:00.907 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T06:13:00.907 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:00.907 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T06:13:00.981 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:00.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T06:13:01.016 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T06:13:01.016 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:01.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T06:13:01.096 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T06:13:01.206 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:01.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T06:13:01.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T06:13:01.301 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T06:13:01.302 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:01.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T06:13:01.377 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T06:13:01.484 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:01.484 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T06:13:01.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T06:13:01.578 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T06:13:01.578 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:01.579 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T06:13:01.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T06:13:01.758 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:01.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T06:13:01.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T06:13:01.851 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T06:13:01.934 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T06:13:01.935 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /var/log/ceph 2024-03-27T06:13:01.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-03-27T06:13:02.001 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T06:13:02.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T06:13:02.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T06:13:02.066 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T06:13:02.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T06:13:02.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T06:13:02.140 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-27T06:13:02.140 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-27T06:13:02.140 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-27T06:13:02.140 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:02.140 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-27T06:13:02.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:02.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:02.452 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:02.453 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-27T06:13:02.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:02.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:02.620 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T06:13:02.620 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-27T06:13:02.620 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580', 'branch': 'wip-rishabh-testing-20240326.131558', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7625580', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240326.131558'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240326.131558', 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 6931, 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0', 'sleep_before_teardown': 0, 'subset': '80/512', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240326.131558', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlC8b1Xb9cMub9U5aw87k8Okmsf9s9uwlDii5o/NuWDK9hG4g1SdEHKoONFWWqP1X63Vx69lC4WeZYVst7rfoI=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODXlBN77fBLoFuEWP82nY+5NNH+IIIOpJ3kkasqacKGnvNe23UPM31XQt76Zglt/8nIOdWIZthrOyatmFeLyeI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '155268c4e432a12433aa833f174f9fe3b1016ae0', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_05:27:11', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T06:13:02.621 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T06:13:02.631 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T06:13:02.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T06:13:02.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T06:13:02.785 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:02.785 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T06:13:02.823 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T06:13:02.823 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-03-27T06:13:02.900 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T06:13:02.900 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:02.900 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-27T06:13:02.900 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:02.900 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:02.901 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-27 06:12:51.968673209 +0000 2024-03-27T06:13:02.901 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-27 06:10:15.039801603 +0000 2024-03-27T06:13:02.901 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-27 06:10:15.039801603 +0000 2024-03-27T06:13:02.901 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-27 06:10:15.039801603 +0000 2024-03-27T06:13:02.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T06:13:03.005 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-27T06:13:03.005 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-27T06:13:03.005 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000676949 s, 756 kB/s 2024-03-27T06:13:03.007 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T06:13:03.044 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-03-27T06:13:03.121 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T06:13:03.121 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:03.121 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-27 06:12:51.969673183 +0000 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-27 06:10:15.611786424 +0000 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-27 06:10:15.611786424 +0000 2024-03-27T06:13:03.122 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-27 06:10:15.611786424 +0000 2024-03-27T06:13:03.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T06:13:03.226 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-27T06:13:03.227 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-27T06:13:03.227 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000308805 s, 1.7 MB/s 2024-03-27T06:13:03.229 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T06:13:03.266 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-27 06:12:51.970673157 +0000 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-27 06:10:16.177771405 +0000 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-27 06:10:16.177771405 +0000 2024-03-27T06:13:03.359 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-27 06:10:16.177771405 +0000 2024-03-27T06:13:03.359 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T06:13:03.437 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-27T06:13:03.437 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-27T06:13:03.438 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000324232 s, 1.6 MB/s 2024-03-27T06:13:03.440 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T06:13:03.489 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-27 06:12:51.970673157 +0000 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-27 06:10:16.742756412 +0000 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-27 06:10:16.742756412 +0000 2024-03-27T06:13:03.535 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-27 06:10:16.742756412 +0000 2024-03-27T06:13:03.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T06:13:03.668 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-27T06:13:03.668 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-27T06:13:03.668 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000320925 s, 1.6 MB/s 2024-03-27T06:13:03.672 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T06:13:03.726 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:03.726 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:03.726 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T06:13:03.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T06:13:03.744 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-03-27T06:13:03.802 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-27 06:12:44.868223549 +0000 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-27 06:07:49.151184811 +0000 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-27 06:07:49.151184811 +0000 2024-03-27T06:13:03.803 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-27 06:07:49.151184811 +0000 2024-03-27T06:13:03.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T06:13:03.871 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-27T06:13:03.872 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-27T06:13:03.872 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000211989 s, 2.4 MB/s 2024-03-27T06:13:03.873 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T06:13:03.931 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-27 06:12:44.868223549 +0000 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-27 06:07:49.496176551 +0000 2024-03-27T06:13:03.987 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-27 06:07:49.496176551 +0000 2024-03-27T06:13:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-27 06:07:49.496176551 +0000 2024-03-27T06:13:03.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T06:13:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-27T06:13:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-27T06:13:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000250376 s, 2.0 MB/s 2024-03-27T06:13:04.056 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T06:13:04.112 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-03-27T06:13:04.168 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-27 06:12:44.869223526 +0000 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-27 06:07:49.837168387 +0000 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-27 06:07:49.837168387 +0000 2024-03-27T06:13:04.169 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-27 06:07:49.837168387 +0000 2024-03-27T06:13:04.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T06:13:04.235 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-27T06:13:04.235 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-27T06:13:04.235 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000215385 s, 2.4 MB/s 2024-03-27T06:13:04.236 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T06:13:04.293 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-03-27T06:13:04.347 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-27 06:12:44.869223526 +0000 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-27 06:07:50.174160319 +0000 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-27 06:07:50.174160319 +0000 2024-03-27T06:13:04.348 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-27 06:07:50.174160319 +0000 2024-03-27T06:13:04.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T06:13:04.413 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-27T06:13:04.413 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-27T06:13:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000203767 s, 2.5 MB/s 2024-03-27T06:13:04.414 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T06:13:04.471 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:04.471 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T06:13:04.471 INFO:tasks.ceph:Generating config... 2024-03-27T06:13:04.472 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-27T06:13:04.472 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-03-27T06:13:04.472 INFO:tasks.ceph:[client] debug client = 20 2024-03-27T06:13:04.472 INFO:tasks.ceph:[client] debug ms = 1 2024-03-27T06:13:04.473 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-27T06:13:04.473 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-27T06:13:04.473 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2024-03-27T06:13:04.473 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-27T06:13:04.474 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-27T06:13:04.475 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-27T06:13:04.475 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-27T06:13:04.476 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-27T06:13:04.476 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-27T06:13:04.476 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T06:13:04.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T06:13:04.602 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T06:13:04.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T06:13:04.733 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T06:13:04.823 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.132'), ('mon.b', '172.21.15.134'), ('mon.c', '[v2:172.21.15.134:3301,v1:172.21.15.134:6790]')] 2024-03-27T06:13:04.824 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.132,172.21.15.134,[v2:172.21.15.134:3301,v1:172.21.15.134:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-27T06:13:04.825 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:04.825 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T06:13:04.879 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.132 --add b 172.21.15.134 --addv c '[v2:172.21.15.134:3301,v1:172.21.15.134:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:05.032 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:05.032 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: generated fsid 3dfa4fc8-5726-4b88-a1bf-7da524e3644e 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = quincy 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:epoch 0 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:fsid 3dfa4fc8-5726-4b88-a1bf-7da524e3644e 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:last_changed 2024-03-27T06:13:05.042609+0000 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:created 2024-03-27T06:13:05.042609+0000 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:min_mon_release 17 (quincy) 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:election_strategy: 1 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:2: [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] mon.c 2024-03-27T06:13:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-27T06:13:05.036 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T06:13:05.084 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3dfa4fc8-5726-4b88-a1bf-7da524e3644e... 2024-03-27T06:13:05.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T06:13:05.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-03-27T06:13:05.145 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-03-27T06:13:05.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = "172.21.15.132,172.21.15.134,[v2:172.21.15.134:3301,v1:172.21.15.134:6790]" 2024-03-27T06:13:05.147 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on skipped message = False 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 3dfa4fc8-5726-4b88-a1bf-7da524e3644e 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-03-27T06:13:05.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-03-27T06:13:05.149 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression mode = aggressive 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 20 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 20 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 10 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd blocked scrub grace period = 3600 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd max markdown count = 1000 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd op complaint time = 180 2024-03-27T06:13:05.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug client = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-03-27T06:13:05.151 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T06:13:05.152 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon down mkfs grace = 300 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon op complaint time = 120 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T06:13:05.153 INFO:teuthology.orchestra.run.smithi134.stdout: client mount timeout = 600 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout: client_shutdown_timeout = 120 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug client = 20 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout: rados mon op timeout = 900 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout: rados osd op timeout = 900 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.b] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.c] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.a] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.c] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.e] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.b] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.d] 2024-03-27T06:13:05.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.f] 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout:[mds] 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: debug mds = 20 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: debug mds balancer = 20 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds bal fragment size max = 10000 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds bal merge size = 5 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds bal split bits = 3 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds bal split size = 100 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds debug frag = True 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds debug scatterstat = True 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds oft prefetch dirfrags = False 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds op complaint time = 180 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds verify scatter = True 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds_dir_prefetch = True 2024-03-27T06:13:05.155 INFO:teuthology.orchestra.run.smithi134.stdout: mds_sleep_rank_change = 5000000.0 2024-03-27T06:13:05.156 INFO:teuthology.orchestra.run.smithi134.stdout: osd op complaint time = 180 2024-03-27T06:13:05.156 INFO:teuthology.orchestra.run.smithi134.stdout: rados mon op timeout = 900 2024-03-27T06:13:05.156 INFO:teuthology.orchestra.run.smithi134.stdout: rados osd op timeout = 900 2024-03-27T06:13:05.248 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.249 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-03-27T06:13:05.250 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = "172.21.15.132,172.21.15.134,[v2:172.21.15.134:3301,v1:172.21.15.134:6790]" 2024-03-27T06:13:05.251 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on skipped message = False 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 3dfa4fc8-5726-4b88-a1bf-7da524e3644e 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-03-27T06:13:05.252 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug trim objects = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-03-27T06:13:05.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore block size = 96636764160 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore compression mode = aggressive 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore fsck on mount = True 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluefs = 20 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluestore = 20 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: debug rocksdb = 10 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd full ratio = 0.9 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd nearfull ratio = 0.8 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: osd blocked scrub grace period = 3600 2024-03-27T06:13:05.254 INFO:teuthology.orchestra.run.smithi132.stdout: osd failsafe full ratio = 0.95 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: osd max markdown count = 1000 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: osd objectstore = bluestore 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: osd op complaint time = 180 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: debug client = 20 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.255 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T06:13:05.256 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon down mkfs grace = 300 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: mon op complaint time = 120 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-03-27T06:13:05.257 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: client mount timeout = 600 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: client_shutdown_timeout = 120 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: debug client = 20 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: rados mon op timeout = 900 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: rados osd op timeout = 900 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.b] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.c] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.a] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.c] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.e] 2024-03-27T06:13:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.b] 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.d] 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.f] 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout:[mds] 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: debug mds = 20 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: debug mds balancer = 20 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds bal fragment size max = 10000 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds bal merge size = 5 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds bal split bits = 3 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds bal split size = 100 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds debug frag = True 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds debug scatterstat = True 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds oft prefetch dirfrags = False 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds op complaint time = 180 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds verify scatter = True 2024-03-27T06:13:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: mds_dir_prefetch = True 2024-03-27T06:13:05.260 INFO:teuthology.orchestra.run.smithi132.stdout: mds_sleep_rank_change = 5000000.0 2024-03-27T06:13:05.260 INFO:teuthology.orchestra.run.smithi132.stdout: osd op complaint time = 180 2024-03-27T06:13:05.260 INFO:teuthology.orchestra.run.smithi132.stdout: rados mon op timeout = 900 2024-03-27T06:13:05.260 INFO:teuthology.orchestra.run.smithi132.stdout: rados osd op timeout = 900 2024-03-27T06:13:05.293 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T06:13:05.293 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T06:13:05.437 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T06:13:05.437 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:05.437 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T06:13:05.485 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:05.485 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T06:13:05.580 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:13:05.580 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:05.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T06:13:05.580 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T06:13:05.724 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:05.725 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:05.820 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:13:05.820 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:05.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T06:13:05.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T06:13:05.865 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:05.865 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:05.922 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T06:13:05.922 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T06:13:05.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T06:13:06.105 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T06:13:06.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T06:13:06.188 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T06:13:06.190 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T06:13:06.191 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-27T06:13:06.371 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-27T06:13:06.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-27T06:13:06.473 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-27T06:13:06.647 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-27T06:13:06.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-27T06:13:06.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-27T06:13:06.935 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-27T06:13:06.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-27T06:13:07.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-27T06:13:07.095 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-27T06:13:07.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-27T06:13:07.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-27T06:13:07.223 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-27T06:13:07.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-27T06:13:07.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-27T06:13:07.347 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-27T06:13:07.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-27T06:13:07.375 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T06:13:07.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T06:13:07.505 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T06:13:07.567 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T06:13:07.567 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T06:13:07.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T06:13:07.660 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:07.660 INFO:tasks.ceph:role: osd.0 2024-03-27T06:13:07.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:13:07.660 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T06:13:07.761 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:07.762 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:08.267 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-27T06:13:08.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-27T06:13:08.273 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T06:13:08.461 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T06:13:08.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T06:13:08.661 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:08.661 INFO:tasks.ceph:role: osd.1 2024-03-27T06:13:08.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:13:08.661 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T06:13:08.757 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:08.758 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:09.233 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-27T06:13:09.240 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-27T06:13:09.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T06:13:09.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T06:13:09.467 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T06:13:09.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:09.557 INFO:tasks.ceph:role: osd.2 2024-03-27T06:13:09.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:13:09.557 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:09.654 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:09.655 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:10.186 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-27T06:13:10.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-27T06:13:10.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-27T06:13:10.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T06:13:10.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-27T06:13:10.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:10.557 INFO:tasks.ceph:role: osd.3 2024-03-27T06:13:10.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:13:10.558 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T06:13:10.655 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:10.655 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:10.655 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:10.656 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:10.656 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:10.656 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:10.656 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:11.100 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-27T06:13:11.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-27T06:13:11.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-27T06:13:11.253 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-27T06:13:11.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:11.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.519+0000 7f946459a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-27T06:13:11.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.521+0000 7f946459a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-27T06:13:11.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.521+0000 7f946459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T06:13:11.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.521+0000 7f946459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T06:13:11.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.521+0000 7f946459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T06:13:11.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:11.528+0000 7f946459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-27T06:13:14.706 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T06:13:14.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:14.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.943+0000 7fc358411640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-27T06:13:14.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.943+0000 7fc358411640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-27T06:13:14.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.943+0000 7fc358411640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T06:13:14.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.943+0000 7fc358411640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T06:13:14.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.943+0000 7fc358411640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T06:13:14.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:14.945+0000 7fc358411640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-27T06:13:17.857 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T06:13:17.950 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:18.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.097+0000 7f38b738c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-27T06:13:18.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.097+0000 7f38b738c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-27T06:13:18.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.097+0000 7f38b738c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T06:13:18.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.097+0000 7f38b738c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T06:13:18.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.098+0000 7f38b738c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T06:13:18.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:18.099+0000 7f38b738c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-27T06:13:21.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T06:13:21.118 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:21.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.266+0000 7f560ccc8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-27T06:13:21.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.266+0000 7f560ccc8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-27T06:13:21.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.267+0000 7f560ccc8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T06:13:21.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.267+0000 7f560ccc8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T06:13:21.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.267+0000 7f560ccc8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T06:13:21.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:21.269+0000 7f560ccc8640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-27T06:13:24.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-27T06:13:24.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-27T06:13:24.297 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:24.297 INFO:tasks.ceph:role: osd.4 2024-03-27T06:13:24.298 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:13:24.298 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:24.375 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:24.872 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-27T06:13:24.874 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-27T06:13:24.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-27T06:13:25.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-27T06:13:25.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-27T06:13:25.116 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:25.117 INFO:tasks.ceph:role: osd.5 2024-03-27T06:13:25.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:13:25.117 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T06:13:25.191 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:25.192 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:25.661 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-27T06:13:25.663 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-27T06:13:25.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-27T06:13:25.754 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-27T06:13:25.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-27T06:13:25.852 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:25.852 INFO:tasks.ceph:role: osd.6 2024-03-27T06:13:25.852 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:13:25.852 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T06:13:25.926 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:26.417 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-27T06:13:26.419 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-27T06:13:26.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-27T06:13:26.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-27T06:13:26.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-27T06:13:26.639 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T06:13:26.639 INFO:tasks.ceph:role: osd.7 2024-03-27T06:13:26.639 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:13:26.639 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T06:13:26.714 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-27T06:13:26.715 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T06:13:26.715 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T06:13:26.715 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T06:13:26.715 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T06:13:27.208 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-27T06:13:27.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-27T06:13:27.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-27T06:13:27.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-27T06:13:27.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:27.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.478+0000 7f82834d3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-27T06:13:27.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.478+0000 7f82834d3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-27T06:13:27.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.478+0000 7f82834d3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T06:13:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.478+0000 7f82834d3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T06:13:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.478+0000 7f82834d3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T06:13:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:27.479+0000 7f82834d3640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-27T06:13:30.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-27T06:13:30.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:30.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.440+0000 7f6a209e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-27T06:13:30.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.440+0000 7f6a209e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-27T06:13:30.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.440+0000 7f6a209e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T06:13:30.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.440+0000 7f6a209e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T06:13:30.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.440+0000 7f6a209e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T06:13:30.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:30.441+0000 7f6a209e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-27T06:13:32.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:32.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:13:33.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-27T06:13:33.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:33.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.403+0000 7fe3cdb73640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-27T06:13:33.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.403+0000 7fe3cdb73640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-27T06:13:33.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.403+0000 7fe3cdb73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T06:13:33.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.403+0000 7fe3cdb73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T06:13:33.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.403+0000 7fe3cdb73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T06:13:33.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:33.404+0000 7fe3cdb73640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-27T06:13:36.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-27T06:13:36.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.384+0000 7fb46060e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.385+0000 7fb46060e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.385+0000 7fb46060e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.385+0000 7fb46060e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.385+0000 7fb46060e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T06:13:36.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:36.385+0000 7fb46060e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-27T06:13:39.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-27T06:13:39.259 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T06:13:39.259 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-27T06:13:39.366 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-27T06:13:39.455 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.455 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-27T06:13:39.593 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-27T06:13:39.688 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T06:13:39.770 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T06:13:39.908 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:39.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T06:13:40.004 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:40.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-27T06:13:40.091 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T06:13:40.122 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-27T06:13:40.193 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-27T06:13:40.262 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-27T06:13:40.330 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-27T06:13:40.399 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-27T06:13:40.468 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-27T06:13:40.535 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:40.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-27T06:13:40.602 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:40.602 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T06:13:40.658 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T06:13:40.658 DEBUG:teuthology.orchestra.run.smithi132:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T06:13:40.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T06:13:40.688 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.y] 2024-03-27T06:13:40.688 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDyuANmxjngBhAAllLhrRZjcLO3abz+ZnbA6w== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.a] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDyuANmFJvCFhAAt7T1PArMiV1YqUsegpbZ6g== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.c] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDyuANmVco2JxAAFCD2ASm/E1tSECjqNgYt4g== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.e] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDyuANmSvVfOBAAfECicUdtKuOZErJHPZu1bQ== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD3uANmDg3/HhAAqWvIyHK143sGzwYniUzV4g== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD6uANmIz1GOBAA96QfINvX9ZkHOXrogvFAKg== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD+uANmZLDqBRAA/PQZz5daT5lDzzqJ1nR3Hw== 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.3] 2024-03-27T06:13:40.689 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQABuQNm6jP/DxAA1lgOvfo5pWpm98cjxZqyvQ== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDyuANmjmDQCxAAKKfkf8AqJDPGXZY7uwItQQ== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.b] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDzuANm7HJPBhAATQPCjtzY5JGm641zkIhKyA== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.d] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDzuANmm5fpDRAAjArA9jaBRRg/3JQh1UXaHA== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.f] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDzuANm7wRJFRAArdTwxUzdahyJrma1Y1J5lw== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.4] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAHuQNmMYWSHBAAi9z4rVWt0BcwccbKbzs5PA== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.5] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAKuQNmbsVPGhAA6s+RULTZ/CB/gXMWVbJjJA== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.6] 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQANuQNmm00aGBAAYYfOuL0LG5J2MVqoZGM4JQ== 2024-03-27T06:13:40.690 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.7] 2024-03-27T06:13:40.691 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAQuQNme/j+FhAAY7TjQsALcwZIuocRW1/1fQ== 2024-03-27T06:13:40.691 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-03-27T06:13:40.691 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDzuANmXvy8HhAAbbiTGKAINAk9wdLazC6dcQ== 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.y] 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDyuANmxjngBhAAllLhrRZjcLO3abz+ZnbA6w== 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.a] 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDyuANmFJvCFhAAt7T1PArMiV1YqUsegpbZ6g== 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.c] 2024-03-27T06:13:40.743 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDyuANmVco2JxAAFCD2ASm/E1tSECjqNgYt4g== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.e] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDyuANmSvVfOBAAfECicUdtKuOZErJHPZu1bQ== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQD3uANmDg3/HhAAqWvIyHK143sGzwYniUzV4g== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQD6uANmIz1GOBAA96QfINvX9ZkHOXrogvFAKg== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQD+uANmZLDqBRAA/PQZz5daT5lDzzqJ1nR3Hw== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.3] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQABuQNm6jP/DxAA1lgOvfo5pWpm98cjxZqyvQ== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.x] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDyuANmjmDQCxAAKKfkf8AqJDPGXZY7uwItQQ== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.b] 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDzuANm7HJPBhAATQPCjtzY5JGm641zkIhKyA== 2024-03-27T06:13:40.744 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.d] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDzuANmm5fpDRAAjArA9jaBRRg/3JQh1UXaHA== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[mds.f] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDzuANm7wRJFRAArdTwxUzdahyJrma1Y1J5lw== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.4] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAHuQNmMYWSHBAAi9z4rVWt0BcwccbKbzs5PA== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.5] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAKuQNmbsVPGhAA6s+RULTZ/CB/gXMWVbJjJA== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.6] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQANuQNmm00aGBAAYYfOuL0LG5J2MVqoZGM4JQ== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.7] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAQuQNme/j+FhAAY7TjQsALcwZIuocRW1/1fQ== 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-03-27T06:13:40.745 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQDzuANmXvy8HhAAbbiTGKAINAk9wdLazC6dcQ== 2024-03-27T06:13:40.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T06:13:40.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T06:13:40.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:40.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:41.075 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:41.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:41.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:41.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:41.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.496 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.770 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:41.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T06:13:41.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T06:13:42.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-27T06:13:42.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:42.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T06:13:43.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T06:13:43.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T06:13:43.178 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T06:13:43.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T06:13:43.267 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T06:13:43.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T06:13:43.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-27T06:13:43.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T06:13:43.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-27T06:13:43.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-27T06:13:43.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T06:13:43.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-27T06:13:43.843 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:43.850 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T06:13:43.899 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T06:13:43.899 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T06:13:43.900 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T06:13:43.904 INFO:tasks.ceph.mon.a:Started 2024-03-27T06:13:43.904 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-27T06:13:43.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-27T06:13:43.939 INFO:tasks.ceph.mon.b:Started 2024-03-27T06:13:43.939 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-27T06:13:43.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-27T06:13:43.941 INFO:tasks.ceph.mon.c:Started 2024-03-27T06:13:43.941 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T06:13:43.941 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-27T06:13:43.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-27T06:13:43.946 INFO:tasks.ceph.mgr.y:Started 2024-03-27T06:13:43.946 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T06:13:43.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T06:13:43.948 INFO:tasks.ceph.mgr.x:Started 2024-03-27T06:13:43.948 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T06:13:43.949 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T06:13:44.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.134+0000 7f7cacc00640 1 Processor -- start 2024-03-27T06:13:44.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.134+0000 7f7cacc00640 1 -- start start 2024-03-27T06:13:44.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.135+0000 7f7cacc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.135+0000 7f7cacc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ca806d440 con 0x7f7ca80cd710 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.135+0000 7f7cacc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ca815b8e0 con 0x7f7ca8154db0 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.135+0000 7f7cacc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca80cdb50 con 0x7f7ca80cbf40 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.135+0000 7f7ca6800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:46878/0 (socket says 172.21.15.132:46878) 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 -- 172.21.15.132:0/642134762 learned_addr learned my addr 172.21.15.132:0/642134762 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 msgr2=0x7f7ca8166220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 msgr2=0x7f7ca8166220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T06:13:44.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.136+0000 7f7ca6800640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.146+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3505038735 0 0) 0x7f7ca815b8e0 con 0x7f7ca8154db0 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.146+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c8c003520 con 0x7f7ca8154db0 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3888137694 0 0) 0x7f7c8c003520 con 0x7f7ca8154db0 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 msgr2=0x7f7ca8166220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7ca8166220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 >> v1:172.21.15.132:6789/0 conn(0x7f7ca80cd710 legacy=0x7f7ca806c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca80cac20 con 0x7f7ca8154db0 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4217832196 0 0) 0x7f7c98001670 con 0x7f7ca8154db0 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 >> v1:172.21.15.134:6789/0 conn(0x7f7ca8154db0 legacy=0x7f7ca80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7c8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c004800 0x7f7c8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7c8c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca80cdb50 con 0x7f7c8c004800 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c00bb00 con 0x7f7c8c0039b0 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c00bd00 con 0x7f7ca80cbf40 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7c8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.147+0000 7f7ca5400640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c004800 0x7f7c8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca6800640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7c8c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca5e00640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 msgr2=0x7f7c8c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:44.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca80cbf40 0x7f7c8c0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca5e00640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c004800 msgr2=0x7f7c8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:44.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c004800 0x7f7c8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.148+0000 7f7ca5e00640 1 -- 172.21.15.132:0/642134762 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c003520 con 0x7f7c8c0039b0 2024-03-27T06:13:44.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.149+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7c8c003dd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800c110 tx=0x7f7c8c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cff7a5afd763e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:44.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.149+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98007c40 con 0x7f7c8c0039b0 2024-03-27T06:13:44.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.149+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c98030070 con 0x7f7c8c0039b0 2024-03-27T06:13:44.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.149+0000 7f7ca4a00640 1 -- 172.21.15.132:0/642134762 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9802ccf0 con 0x7f7c8c0039b0 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.150+0000 7f7cacc00640 1 -- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 msgr2=0x7f7c8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.150+0000 7f7cacc00640 1 --2- 172.21.15.132:0/642134762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7c8c003dd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800c110 tx=0x7f7c8c005830 comp rx=0 tx=0).stop 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.150+0000 7f7ca6800640 1 -- 172.21.15.132:0/642134762 reap_dead start 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.151+0000 7f7cacc00640 1 -- 172.21.15.132:0/642134762 shutdown_connections 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.151+0000 7f7cacc00640 1 -- 172.21.15.132:0/642134762 >> 172.21.15.132:0/642134762 conn(0x7f7ca80ccbf0 msgr2=0x7f7ca80cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.151+0000 7f7cacc00640 1 -- 172.21.15.132:0/642134762 shutdown_connections 2024-03-27T06:13:44.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.151+0000 7f7cacc00640 1 -- 172.21.15.132:0/642134762 wait complete. 2024-03-27T06:13:44.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.154+0000 7f7cacc00640 1 Processor -- start 2024-03-27T06:13:44.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.154+0000 7f7cacc00640 1 -- start start 2024-03-27T06:13:44.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7ca80c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ca8153940 0x7f7ca81e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca80cc860 con 0x7f7ca8153940 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca80caf50 con 0x7f7c8c0039b0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7cacc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca815b600 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7ca5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7ca5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:46916/0 (socket says 172.21.15.132:46916) 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7ca5400640 1 -- 172.21.15.132:0/2100315590 learned_addr learned my addr 172.21.15.132:0/2100315590 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7ca80c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.155+0000 7f7ca6800640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ca8153940 0x7f7ca81e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 -- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 msgr2=0x7f7ca80c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7ca80c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 -- 172.21.15.132:0/2100315590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ca8153940 msgr2=0x7f7ca81e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ca8153940 0x7f7ca81e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 -- 172.21.15.132:0/2100315590 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca814d200 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5e00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7ca80c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:13:44.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7ca5400640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400f630 tx=0x7f7c9400faf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:44.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c940052e0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c94018070 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94003830 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca80c4a00 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.156+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81f04d0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.157+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c940039d0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.157+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7c9401dcd0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.158+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca80cc4d0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.160+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7ca80cc4d0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.261+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7ca81effe0 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.265+0000 7f7c9e000640 1 -- 172.21.15.132:0/2100315590 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f7ca806d440 con 0x7f7ca81f0cd0 2024-03-27T06:13:44.256 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-03-27T06:13:44.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.266+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 msgr2=0x7f7ca814ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.266+0000 7f7cacc00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400f630 tx=0x7f7c9400faf0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 shutdown_connections 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ca8153940 0x7f7ca81e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7ca81f0cd0 0x7f7ca814ccc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 --2- 172.21.15.132:0/2100315590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c8c0039b0 0x7f7ca80c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 >> 172.21.15.132:0/2100315590 conn(0x7f7ca80ccbf0 msgr2=0x7f7ca806b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 shutdown_connections 2024-03-27T06:13:44.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:44.267+0000 7f7cacc00640 1 -- 172.21.15.132:0/2100315590 wait complete. 2024-03-27T06:13:44.276 INFO:tasks.ceph:check_enable_crimson: False 2024-03-27T06:13:44.276 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T06:13:44.276 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:44.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T06:13:44.352 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:44.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T06:13:44.477 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:44.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T06:13:44.555 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:13:44.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-27T06:13:44.644 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:44.654+0000 7f830ec49040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T06:13:44.686 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:44.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-27T06:13:44.699 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:44.708+0000 7f830ec49040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T06:13:44.713 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:44.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-27T06:13:44.724 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:44.733+0000 7f2b70626040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T06:13:44.780 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:44.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-27T06:13:44.785 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:44.794+0000 7f2b70626040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T06:13:44.847 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-27T06:13:44.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-27T06:13:44.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 804650a8-bce8-4db0-87fc-a48dafbb9039 0 2024-03-27T06:13:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.064+0000 7f41e6e95640 1 Processor -- start 2024-03-27T06:13:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.064+0000 7f41e6e95640 1 -- start start 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f41e00c3db0 0x7f41e006be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f41e015aab0 con 0x7f41e00c3db0 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f41e015ad50 con 0x7f41e00cd540 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e6e95640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f41e006c3d0 con 0x7f41e015e870 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e5432640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47816/0 (socket says 172.21.15.134:47816) 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e5432640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35960/0 (socket says 172.21.15.134:35960) 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 -- 172.21.15.134:0/1778159379 learned_addr learned my addr 172.21.15.134:0/1778159379 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 -- 172.21.15.134:0/1778159379 >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 msgr2=0x7f41e00c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 --2- 172.21.15.134:0/1778159379 >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 -- 172.21.15.134:0/1778159379 >> v2:172.21.15.132:3300/0 conn(0x7f41e00c3db0 msgr2=0x7f41e006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 --2- 172.21.15.134:0/1778159379 >> v2:172.21.15.132:3300/0 conn(0x7f41e00c3db0 0x7f41e006be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41dffff640 1 -- 172.21.15.134:0/1778159379 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e015b010 con 0x7f41e015e870 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.065+0000 7f41e5432640 1 --2- 172.21.15.134:0/1778159379 >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:45.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.066+0000 7f41dffff640 1 --2- 172.21.15.134:0/1778159379 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41e0071bd0 tx=0x7f41cc02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5215cf14e346f5de server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.066+0000 7f41df7fe640 1 -- 172.21.15.134:0/1778159379 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc035020 con 0x7f41e015e870 2024-03-27T06:13:45.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.066+0000 7f41df7fe640 1 -- 172.21.15.134:0/1778159379 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41cc02d040 con 0x7f41e015e870 2024-03-27T06:13:45.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.066+0000 7f41df7fe640 1 -- 172.21.15.134:0/1778159379 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc031a80 con 0x7f41e015e870 2024-03-27T06:13:45.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 -- 172.21.15.134:0/1778159379 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 msgr2=0x7f41e00cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 --2- 172.21.15.134:0/1778159379 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41e0071bd0 tx=0x7f41cc02b900 comp rx=0 tx=0).stop 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 -- 172.21.15.134:0/1778159379 shutdown_connections 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 --2- 172.21.15.134:0/1778159379 >> v2:172.21.15.134:3300/0 conn(0x7f41e00cd540 0x7f41e00c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 --2- 172.21.15.134:0/1778159379 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e00cd000 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 --2- 172.21.15.134:0/1778159379 >> v2:172.21.15.132:3300/0 conn(0x7f41e00c3db0 0x7f41e006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 -- 172.21.15.134:0/1778159379 >> 172.21.15.134:0/1778159379 conn(0x7f41e006c630 msgr2=0x7f41e006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 -- 172.21.15.134:0/1778159379 shutdown_connections 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.067+0000 7f41e6e95640 1 -- 172.21.15.134:0/1778159379 wait complete. 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.068+0000 7f41e6e95640 1 Processor -- start 2024-03-27T06:13:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.068+0000 7f41e6e95640 1 -- start start 2024-03-27T06:13:45.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 0x7f41e00c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 0x7f41e0150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41e015ec90 con 0x7f41e00cd540 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41e015b5d0 con 0x7f41e00c3db0 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e6e95640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f41e015ee10 con 0x7f41e015e870 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 0x7f41e00c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e5432640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e5432640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47822/0 (socket says 172.21.15.134:47822) 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e5432640 1 -- 172.21.15.134:0/4187127696 learned_addr learned my addr 172.21.15.134:0/4187127696 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 -- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 msgr2=0x7f41e014d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 -- 172.21.15.134:0/4187127696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 msgr2=0x7f41e0150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41dffff640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 0x7f41e0150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 0x7f41e0150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 -- 172.21.15.134:0/4187127696 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e0159b20 con 0x7f41e00c3db0 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e5432640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41dffff640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 0x7f41e0150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:13:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.069+0000 7f41e4c31640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 0x7f41e00c2c20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41d400b410 tx=0x7f41d400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4015020 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d4011850 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e01f1620 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4011c40 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e01f0b40 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f41d4018020 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.070+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f41d40196c0 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.071+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e00cba70 con 0x7f41e00c3db0 2024-03-27T06:13:45.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.073+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f41e00cba70 con 0x7f41e00c3db0 2024-03-27T06:13:45.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.147+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "804650a8-bce8-4db0-87fc-a48dafbb9039", "id": 0} v 0) -- 0x7f41e0068950 con 0x7f41e00c3db0 2024-03-27T06:13:45.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.153+0000 7f41dd7fa640 1 -- 172.21.15.134:0/4187127696 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "804650a8-bce8-4db0-87fc-a48dafbb9039", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f41d401f020 con 0x7f41e00c3db0 2024-03-27T06:13:45.144 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-03-27T06:13:45.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 msgr2=0x7f41e00c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 0x7f41e00c2c20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41d400b410 tx=0x7f41d400b8d0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 shutdown_connections 2024-03-27T06:13:45.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41e015e870 0x7f41e014d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00cd540 0x7f41e0150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 --2- 172.21.15.134:0/4187127696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41e00c3db0 0x7f41e00c2c20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 >> 172.21.15.134:0/4187127696 conn(0x7f41e006c630 msgr2=0x7f41e006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 shutdown_connections 2024-03-27T06:13:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.154+0000 7f41e6e95640 1 -- 172.21.15.134:0/4187127696 wait complete. 2024-03-27T06:13:45.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 7302bf98-cd95-401d-8a7f-c1b37532aa4e 1 2024-03-27T06:13:45.255 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.264+0000 7f830ec49040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T06:13:45.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.265+0000 7f52d8088640 1 Processor -- start 2024-03-27T06:13:45.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.265+0000 7f52d8088640 1 -- start start 2024-03-27T06:13:45.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d8088640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 0x7f52d006b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d8088640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f52d00ca1f0 0x7f52d006bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d8088640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d00c7b10 con 0x7f52d0159430 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d8088640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f52d00c8a00 con 0x7f52d00ca1f0 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d8088640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f52d00c7df0 con 0x7f52d01524b0 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d5e24640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 0x7f52d006b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d5623640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f52d0159430 0x7f52d00c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:46504/0 (socket says 172.21.15.134:46504) 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.266+0000 7f52d5623640 1 -- 172.21.15.134:0/2304480331 learned_addr learned my addr 172.21.15.134:0/2304480331 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d5e24640 1 -- 172.21.15.134:0/2304480331 >> v2:172.21.15.132:3300/0 conn(0x7f52d00ca1f0 msgr2=0x7f52d006bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d5e24640 1 --2- 172.21.15.134:0/2304480331 >> v2:172.21.15.132:3300/0 conn(0x7f52d00ca1f0 0x7f52d006bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d5e24640 1 -- 172.21.15.134:0/2304480331 >> v1:172.21.15.134:6789/0 conn(0x7f52d0159430 legacy=0x7f52d00c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d4e22640 1 -- 172.21.15.134:0/2304480331 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121968134 0 0) 0x7f52d00c7b10 con 0x7f52d0159430 2024-03-27T06:13:45.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d5e24640 1 -- 172.21.15.134:0/2304480331 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d00c80d0 con 0x7f52d01524b0 2024-03-27T06:13:45.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d5e24640 1 --2- 172.21.15.134:0/2304480331 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 0x7f52d006b7d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52c00097a0 tx=0x7f52c002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c15d2b3e921b48d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d4e22640 1 -- 172.21.15.134:0/2304480331 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0034070 con 0x7f52d01524b0 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d4e22640 1 -- 172.21.15.134:0/2304480331 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c0030610 con 0x7f52d01524b0 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.267+0000 7f52d4e22640 1 -- 172.21.15.134:0/2304480331 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c00309e0 con 0x7f52d01524b0 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 -- 172.21.15.134:0/2304480331 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 msgr2=0x7f52d006b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 --2- 172.21.15.134:0/2304480331 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 0x7f52d006b7d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52c00097a0 tx=0x7f52c002b960 comp rx=0 tx=0).stop 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 -- 172.21.15.134:0/2304480331 shutdown_connections 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 --2- 172.21.15.134:0/2304480331 >> v2:172.21.15.132:3300/0 conn(0x7f52d00ca1f0 0x7f52d006bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 --2- 172.21.15.134:0/2304480331 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d01524b0 0x7f52d006b7d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 -- 172.21.15.134:0/2304480331 >> 172.21.15.134:0/2304480331 conn(0x7f52d006c630 msgr2=0x7f52d006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 -- 172.21.15.134:0/2304480331 shutdown_connections 2024-03-27T06:13:45.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.268+0000 7f52d8088640 1 -- 172.21.15.134:0/2304480331 wait complete. 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.269+0000 7f52d8088640 1 Processor -- start 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.269+0000 7f52d8088640 1 -- start start 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52d01524b0 0x7f52d01542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 0x7f52d01556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52d0152890 con 0x7f52d01524b0 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f52d00c8690 con 0x7f52d00ca1f0 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d8088640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f52d0152a10 con 0x7f52d0159430 2024-03-27T06:13:45.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d5e24640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d5e24640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35978/0 (socket says 172.21.15.134:35978) 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d5e24640 1 -- 172.21.15.134:0/1625884990 learned_addr learned my addr 172.21.15.134:0/1625884990 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 0x7f52d01556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 -- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 msgr2=0x7f52d00ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 -- 172.21.15.134:0/1625884990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52d01524b0 msgr2=0x7f52d01542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52d01524b0 0x7f52d01542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d6625640 1 -- 172.21.15.134:0/1625884990 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d01ee670 con 0x7f52d0159430 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.270+0000 7f52d5e24640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52d6625640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 0x7f52d01556a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f52c800be50 tx=0x7f52c800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8019070 con 0x7f52d0159430 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c8014440 con 0x7f52d0159430 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8014830 con 0x7f52d0159430 2024-03-27T06:13:45.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d01ee950 con 0x7f52d0159430 2024-03-27T06:13:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d01eef10 con 0x7f52d0159430 2024-03-27T06:13:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f52c801c070 con 0x7f52d0159430 2024-03-27T06:13:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.271+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f52c801d760 con 0x7f52d0159430 2024-03-27T06:13:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.272+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d0068950 con 0x7f52d0159430 2024-03-27T06:13:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.274+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f52c8023070 con 0x7f52d0159430 2024-03-27T06:13:45.312 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.321+0000 7f830ec49040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T06:13:45.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.347+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7302bf98-cd95-401d-8a7f-c1b37532aa4e", "id": 1} v 0) -- 0x7f52d0068ff0 con 0x7f52d0159430 2024-03-27T06:13:45.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.354+0000 7f52c67fc640 1 -- 172.21.15.134:0/1625884990 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7302bf98-cd95-401d-8a7f-c1b37532aa4e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f52c8014c40 con 0x7f52d0159430 2024-03-27T06:13:45.345 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 msgr2=0x7f52d01556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 0x7f52d01556a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f52c800be50 tx=0x7f52c800bf40 comp rx=0 tx=0).stop 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 shutdown_connections 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f52d0159430 0x7f52d01556a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52d01524b0 0x7f52d01542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 --2- 172.21.15.134:0/1625884990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52d00ca1f0 0x7f52d00ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 >> 172.21.15.134:0/1625884990 conn(0x7f52d006c630 msgr2=0x7f52d0157af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 shutdown_connections 2024-03-27T06:13:45.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.355+0000 7f52d8088640 1 -- 172.21.15.134:0/1625884990 wait complete. 2024-03-27T06:13:45.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new d69ad874-437c-49b7-83ca-d7aef2e59047 2 2024-03-27T06:13:45.360 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.369+0000 7f830ec49040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T06:13:45.387 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.396+0000 7f2b70626040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T06:13:45.416 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.425+0000 7f830ec49040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T06:13:45.451 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.460+0000 7f2b70626040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T06:13:45.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.465+0000 7f9ff88eb640 1 Processor -- start 2024-03-27T06:13:45.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.465+0000 7f9ff88eb640 1 -- start start 2024-03-27T06:13:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff88eb640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 0x7f9ff415b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff88eb640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9ff40cb2f0 0x7f9ff40d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff88eb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff415bb40 con 0x7f9ff40d0740 2024-03-27T06:13:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff88eb640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff4155360 con 0x7f9ff40cb2f0 2024-03-27T06:13:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff88eb640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff40ccd30 con 0x7f9ff4154f80 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff259c640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 0x7f9ff415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff2d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9ff40d0740 0x7f9ff40c8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:46520/0 (socket says 172.21.15.134:46520) 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff2d9d640 1 -- 172.21.15.134:0/2602460359 learned_addr learned my addr 172.21.15.134:0/2602460359 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff159a640 1 -- 172.21.15.134:0/2602460359 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663111391 0 0) 0x7f9ff415bb40 con 0x7f9ff40d0740 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff259c640 1 -- 172.21.15.134:0/2602460359 >> v2:172.21.15.132:3300/0 conn(0x7f9ff40cb2f0 msgr2=0x7f9ff40d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff259c640 1 --2- 172.21.15.134:0/2602460359 >> v2:172.21.15.132:3300/0 conn(0x7f9ff40cb2f0 0x7f9ff40d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff259c640 1 -- 172.21.15.134:0/2602460359 >> v1:172.21.15.134:6789/0 conn(0x7f9ff40d0740 legacy=0x7f9ff40c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.466+0000 7f9ff259c640 1 -- 172.21.15.134:0/2602460359 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff40ccfd0 con 0x7f9ff4154f80 2024-03-27T06:13:45.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.467+0000 7f9ff259c640 1 --2- 172.21.15.134:0/2602460359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 0x7f9ff415b600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40097a0 tx=0x7f9fe402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4087b9ba1b47268e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.467+0000 7f9ff159a640 1 -- 172.21.15.134:0/2602460359 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40315a0 con 0x7f9ff4154f80 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.467+0000 7f9ff159a640 1 -- 172.21.15.134:0/2602460359 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe4037070 con 0x7f9ff4154f80 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.467+0000 7f9ff159a640 1 -- 172.21.15.134:0/2602460359 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4031960 con 0x7f9ff4154f80 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 -- 172.21.15.134:0/2602460359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 msgr2=0x7f9ff415b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/2602460359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 0x7f9ff415b600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40097a0 tx=0x7f9fe402b840 comp rx=0 tx=0).stop 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 -- 172.21.15.134:0/2602460359 shutdown_connections 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/2602460359 >> v2:172.21.15.132:3300/0 conn(0x7f9ff40cb2f0 0x7f9ff40d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/2602460359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff4154f80 0x7f9ff415b600 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 -- 172.21.15.134:0/2602460359 >> 172.21.15.134:0/2602460359 conn(0x7f9ff406d9f0 msgr2=0x7f9ff406de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 -- 172.21.15.134:0/2602460359 shutdown_connections 2024-03-27T06:13:45.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.468+0000 7f9ff88eb640 1 -- 172.21.15.134:0/2602460359 wait complete. 2024-03-27T06:13:45.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.469+0000 7f9ff88eb640 1 Processor -- start 2024-03-27T06:13:45.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.469+0000 7f9ff88eb640 1 -- start start 2024-03-27T06:13:45.461 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.469+0000 7f830ec49040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ff40d0740 0x7f9ff414bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 0x7f9ff415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff40cd870 con 0x7f9ff40d0740 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff40cd590 con 0x7f9ff4154f80 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff88eb640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff40cc0a0 con 0x7f9ff40cb2f0 2024-03-27T06:13:45.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff259c640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 0x7f9ff415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff259c640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47866/0 (socket says 172.21.15.134:47866) 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff259c640 1 -- 172.21.15.134:0/4283656449 learned_addr learned my addr 172.21.15.134:0/4283656449 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 -- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 msgr2=0x7f9ff414daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 -- 172.21.15.134:0/4283656449 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ff40d0740 msgr2=0x7f9ff414bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ff40d0740 0x7f9ff414bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 -- 172.21.15.134:0/4283656449 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff415e0f0 con 0x7f9ff4154f80 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff259c640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9ff2d9d640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 0x7f9ff415c790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c700 tx=0x7f9fec00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec018070 con 0x7f9ff4154f80 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.470+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fec00ce10 con 0x7f9ff4154f80 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00a880 con 0x7f9ff4154f80 2024-03-27T06:13:45.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff414e400 con 0x7f9ff4154f80 2024-03-27T06:13:45.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff414e9c0 con 0x7f9ff4154f80 2024-03-27T06:13:45.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9fec01b070 con 0x7f9ff4154f80 2024-03-27T06:13:45.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f9fec0077b0 con 0x7f9ff4154f80 2024-03-27T06:13:45.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.471+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff40cfff0 con 0x7f9ff4154f80 2024-03-27T06:13:45.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.475+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9ff40cfff0 con 0x7f9ff4154f80 2024-03-27T06:13:45.502 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.511+0000 7f2b70626040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T06:13:45.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.546+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d69ad874-437c-49b7-83ca-d7aef2e59047", "id": 2} v 0) -- 0x7f9ff4068950 con 0x7f9ff4154f80 2024-03-27T06:13:45.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.553+0000 7f9fe2ffd640 1 -- 172.21.15.134:0/4283656449 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d69ad874-437c-49b7-83ca-d7aef2e59047", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f9fec0071c0 con 0x7f9ff4154f80 2024-03-27T06:13:45.544 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-03-27T06:13:45.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 msgr2=0x7f9ff415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 0x7f9ff415c790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c700 tx=0x7f9fec00cbc0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 shutdown_connections 2024-03-27T06:13:45.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4154f80 0x7f9ff415c790 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ff40d0740 0x7f9ff414bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 --2- 172.21.15.134:0/4283656449 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ff40cb2f0 0x7f9ff414daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 >> 172.21.15.134:0/4283656449 conn(0x7f9ff406d9f0 msgr2=0x7f9ff4153790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 shutdown_connections 2024-03-27T06:13:45.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.554+0000 7f9ff88eb640 1 -- 172.21.15.134:0/4283656449 wait complete. 2024-03-27T06:13:45.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new bb044ddc-5a7d-4afd-8ebb-39a2776d3046 3 2024-03-27T06:13:45.565 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.575+0000 7f2b70626040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T06:13:45.582 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.591+0000 7f830ec49040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T06:13:45.615 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.624+0000 7f2b70626040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T06:13:45.656 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.665+0000 7f830ec49040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T06:13:45.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 Processor -- start 2024-03-27T06:13:45.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 -- start start 2024-03-27T06:13:45.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ae406f960 0x7f5ae406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae40ca8c0 con 0x7f5ae406d660 2024-03-27T06:13:45.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae40c8d10 con 0x7f5ae406f960 2024-03-27T06:13:45.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.692+0000 7f5ae89ac640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae415b990 con 0x7f5ae4154d00 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ae406f960 0x7f5ae406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5ae259c640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ae406f960 0x7f5ae406d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35998/0 (socket says 172.21.15.134:35998) 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 -- 172.21.15.134:0/2811339185 learned_addr learned my addr 172.21.15.134:0/2811339185 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5ae259c640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47876/0 (socket says 172.21.15.134:47876) 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 -- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 msgr2=0x7f5ae415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 -- 172.21.15.134:0/2811339185 >> v1:172.21.15.134:6789/0 conn(0x7f5ae406d660 legacy=0x7f5ae40ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5adbfff640 1 -- 172.21.15.134:0/2811339185 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae415b0f0 con 0x7f5ae406f960 2024-03-27T06:13:45.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.693+0000 7f5ae259c640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5adbfff640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406f960 0x7f5ae406d120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009870 tx=0x7f5acc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82339b778b7b6824 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae1d9b640 1 -- 172.21.15.134:0/2811339185 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc035020 con 0x7f5ae406f960 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae1d9b640 1 -- 172.21.15.134:0/2811339185 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5acc031640 con 0x7f5ae406f960 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae1d9b640 1 -- 172.21.15.134:0/2811339185 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc02d040 con 0x7f5ae406f960 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 -- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406f960 msgr2=0x7f5ae406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406f960 0x7f5ae406d120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009870 tx=0x7f5acc02b8f0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 -- 172.21.15.134:0/2811339185 shutdown_connections 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406f960 0x7f5ae406d120 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/2811339185 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae4154d00 0x7f5ae415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.694+0000 7f5ae89ac640 1 -- 172.21.15.134:0/2811339185 >> 172.21.15.134:0/2811339185 conn(0x7f5ae4067460 msgr2=0x7f5ae4067840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.695+0000 7f5ae89ac640 1 -- 172.21.15.134:0/2811339185 shutdown_connections 2024-03-27T06:13:45.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.695+0000 7f5ae89ac640 1 -- 172.21.15.134:0/2811339185 wait complete. 2024-03-27T06:13:45.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.696+0000 7f5ae89ac640 1 Processor -- start 2024-03-27T06:13:45.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.696+0000 7f5ae89ac640 1 -- start start 2024-03-27T06:13:45.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.696+0000 7f5ae89ac640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae406f960 0x7f5ae41f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4154d00 0x7f5ae40c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae415b6b0 con 0x7f5ae4154d00 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae406cc90 con 0x7f5ae406d660 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae406c920 con 0x7f5ae406f960 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae406f960 0x7f5ae41f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36000/0 (socket says 172.21.15.134:36000) 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 -- 172.21.15.134:0/890413636 learned_addr learned my addr 172.21.15.134:0/890413636 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 -- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae406f960 msgr2=0x7f5ae41f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae406f960 0x7f5ae41f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 -- 172.21.15.134:0/890413636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4154d00 msgr2=0x7f5ae40c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4154d00 0x7f5ae40c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 -- 172.21.15.134:0/890413636 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae406ef80 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae259c640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400ec10 tx=0x7f5ad400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4017070 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad400cc30 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae41f4b40 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4013470 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae4155980 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ad401a070 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.697+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f5ad4006020 con 0x7f5ae406d660 2024-03-27T06:13:45.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.698+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae4068950 con 0x7f5ae406d660 2024-03-27T06:13:45.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.700+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5ad4013610 con 0x7f5ae406d660 2024-03-27T06:13:45.705 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.715+0000 7f830ec49040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T06:13:45.745 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.754+0000 7f2b70626040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T06:13:45.759 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.769+0000 7f830ec49040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T06:13:45.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.773+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bb044ddc-5a7d-4afd-8ebb-39a2776d3046", "id": 3} v 0) -- 0x7f5ae406e0f0 con 0x7f5ae406d660 2024-03-27T06:13:45.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.779+0000 7f5adaffd640 1 -- 172.21.15.134:0/890413636 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb044ddc-5a7d-4afd-8ebb-39a2776d3046", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f5ad4013a20 con 0x7f5ae406d660 2024-03-27T06:13:45.770 INFO:teuthology.orchestra.run.smithi134.stdout:3 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 msgr2=0x7f5ae4159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400ec10 tx=0x7f5ad400c520 comp rx=0 tx=0).stop 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 shutdown_connections 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4154d00 0x7f5ae40c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ae406f960 0x7f5ae41f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 --2- 172.21.15.134:0/890413636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ae406d660 0x7f5ae4159d00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 >> 172.21.15.134:0/890413636 conn(0x7f5ae4067460 msgr2=0x7f5ae41527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 shutdown_connections 2024-03-27T06:13:45.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.780+0000 7f5ae89ac640 1 -- 172.21.15.134:0/890413636 wait complete. 2024-03-27T06:13:45.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 5fc09bbc-14fa-4638-8922-8f20d62654b1 4 2024-03-27T06:13:45.810 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:45.819+0000 7f830ec49040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T06:13:45.826 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.836+0000 7f2b70626040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T06:13:45.880 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.889+0000 7f2b70626040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T06:13:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.890+0000 7f2cfc7f7640 1 Processor -- start 2024-03-27T06:13:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.890+0000 7f2cfc7f7640 1 -- start start 2024-03-27T06:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfc7f7640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfc7f7640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2cf40afd90 0x7f2cf40c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfc7f7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf41585b0 con 0x7f2cf40c8040 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfc7f7640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf4151e40 con 0x7f2cf40afd90 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfc7f7640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf4158120 con 0x7f2cf4151a60 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47908/0 (socket says 172.21.15.134:47908) 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 -- 172.21.15.134:0/738858631 learned_addr learned my addr 172.21.15.134:0/738858631 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfad94640 1 --1- 172.21.15.134:0/738858631 >> v1:172.21.15.134:6789/0 conn(0x7f2cf40c8040 0x7f2cf4073040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:46532/0 (socket says 172.21.15.134:46532) 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cf9591640 1 -- 172.21.15.134:0/738858631 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4193081808 0 0) 0x7f2cf41585b0 con 0x7f2cf40c8040 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 -- 172.21.15.134:0/738858631 >> v2:172.21.15.132:3300/0 conn(0x7f2cf40afd90 msgr2=0x7f2cf40c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 --2- 172.21.15.134:0/738858631 >> v2:172.21.15.132:3300/0 conn(0x7f2cf40afd90 0x7f2cf40c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 -- 172.21.15.134:0/738858631 >> v1:172.21.15.134:6789/0 conn(0x7f2cf40c8040 legacy=0x7f2cf4073040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.891+0000 7f2cfa593640 1 -- 172.21.15.134:0/738858631 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf40c9710 con 0x7f2cf4151a60 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.892+0000 7f2cfa593640 1 --2- 172.21.15.134:0/738858631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009870 tx=0x7f2cf002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e28465c33a6c15c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.892+0000 7f2cf9591640 1 -- 172.21.15.134:0/738858631 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0035020 con 0x7f2cf4151a60 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.892+0000 7f2cf9591640 1 -- 172.21.15.134:0/738858631 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf0031640 con 0x7f2cf4151a60 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.892+0000 7f2cf9591640 1 -- 172.21.15.134:0/738858631 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf002d040 con 0x7f2cf4151a60 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/738858631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 msgr2=0x7f2cf406c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/738858631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009870 tx=0x7f2cf002b8f0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/738858631 shutdown_connections 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/738858631 >> v2:172.21.15.132:3300/0 conn(0x7f2cf40afd90 0x7f2cf40c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/738858631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf4151a60 0x7f2cf406c630 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/738858631 >> 172.21.15.134:0/738858631 conn(0x7f2cf40c9c00 msgr2=0x7f2cf40ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/738858631 shutdown_connections 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.893+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/738858631 wait complete. 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.894+0000 7f2cfc7f7640 1 Processor -- start 2024-03-27T06:13:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.894+0000 7f2cfc7f7640 1 -- start start 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2cf40afd90 0x7f2cf40b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cf4151a60 0x7f2cf4156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf4157db0 con 0x7f2cf40afd90 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf4157ad0 con 0x7f2cf4151a60 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf406e9e0 con 0x7f2cf40c8040 2024-03-27T06:13:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfad94640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cf4151a60 0x7f2cf4156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47924/0 (socket says 172.21.15.134:47924) 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 -- 172.21.15.134:0/4039636216 learned_addr learned my addr 172.21.15.134:0/4039636216 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfa593640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2cf40afd90 0x7f2cf40b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 -- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cf4151a60 msgr2=0x7f2cf4156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cf4151a60 0x7f2cf4156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 -- 172.21.15.134:0/4039636216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2cf40afd90 msgr2=0x7f2cf40b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2cf40afd90 0x7f2cf40b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 -- 172.21.15.134:0/4039636216 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf41ee7b0 con 0x7f2cf40c8040 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cf9d92640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400b4d0 tx=0x7f2ce400b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4014070 con 0x7f2cf40c8040 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce4010610 con 0x7f2cf40c8040 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4010a00 con 0x7f2cf40c8040 2024-03-27T06:13:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf41eea90 con 0x7f2cf40c8040 2024-03-27T06:13:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.895+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf41ef330 con 0x7f2cf40c8040 2024-03-27T06:13:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.896+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ce4017070 con 0x7f2cf40c8040 2024-03-27T06:13:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.896+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f2ce4019db0 con 0x7f2cf40c8040 2024-03-27T06:13:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.896+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf4067c90 con 0x7f2cf40c8040 2024-03-27T06:13:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.898+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2ce4010ce0 con 0x7f2cf40c8040 2024-03-27T06:13:45.940 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:45.950+0000 7f2b70626040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T06:13:45.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.970+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5fc09bbc-14fa-4638-8922-8f20d62654b1", "id": 4} v 0) -- 0x7f2cf4068950 con 0x7f2cf40c8040 2024-03-27T06:13:45.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.976+0000 7f2ceaffd640 1 -- 172.21.15.134:0/4039636216 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5fc09bbc-14fa-4638-8922-8f20d62654b1", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f2ce4017430 con 0x7f2cf40c8040 2024-03-27T06:13:45.967 INFO:teuthology.orchestra.run.smithi134.stdout:4 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 msgr2=0x7f2cf40c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400b4d0 tx=0x7f2ce400b990 comp rx=0 tx=0).stop 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 shutdown_connections 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cf4151a60 0x7f2cf4156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2cf40c8040 0x7f2cf40c6980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 --2- 172.21.15.134:0/4039636216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2cf40afd90 0x7f2cf40b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 >> 172.21.15.134:0/4039636216 conn(0x7f2cf40c9c00 msgr2=0x7f2cf414fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:45.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 shutdown_connections 2024-03-27T06:13:45.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:45.977+0000 7f2cfc7f7640 1 -- 172.21.15.134:0/4039636216 wait complete. 2024-03-27T06:13:45.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new ed2023b3-8f13-4157-ba83-e8245614800c 5 2024-03-27T06:13:45.997 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:46.006+0000 7f2b70626040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T06:13:46.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.129+0000 7fcf684b5640 1 Processor -- start 2024-03-27T06:13:46.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.129+0000 7fcf684b5640 1 -- start start 2024-03-27T06:13:46.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf684b5640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fcf60159430 0x7fcf600c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf684b5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf600c7b10 con 0x7fcf60152890 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf684b5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf600c7df0 con 0x7fcf601524b0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf684b5640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fcf600c8a00 con 0x7fcf60159430 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf66251640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcf601524b0 0x7fcf6006b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:46542/0 (socket says 172.21.15.134:46542) 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf66251640 1 -- 172.21.15.134:0/4176653141 learned_addr learned my addr 172.21.15.134:0/4176653141 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2784706937 0 0) 0x7fcf600c7df0 con 0x7fcf601524b0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf4c003520 con 0x7fcf601524b0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf65a50640 1 --2- 172.21.15.134:0/4176653141 >> v2:172.21.15.132:3300/0 conn(0x7fcf60159430 0x7fcf600c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1790998831 0 0) 0x7fcf4c003520 con 0x7fcf601524b0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 >> v2:172.21.15.132:3300/0 conn(0x7fcf60159430 msgr2=0x7fcf600c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 --2- 172.21.15.134:0/4176653141 >> v2:172.21.15.132:3300/0 conn(0x7fcf60159430 0x7fcf600c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 >> v1:172.21.15.132:6789/0 conn(0x7fcf60152890 legacy=0x7fcf600cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf600c80d0 con 0x7fcf601524b0 2024-03-27T06:13:46.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4217832196 0 0) 0x7fcf50001670 con 0x7fcf601524b0 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 >> v1:172.21.15.134:6789/0 conn(0x7fcf601524b0 legacy=0x7fcf6006b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf4c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf4c004800 0x7fcf4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf4c005210 0x7fcf4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf600c8a00 con 0x7fcf4c004800 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c00c120 con 0x7fcf4c003a00 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c00c320 con 0x7fcf4c005210 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.130+0000 7fcf66251640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf4c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66a52640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf4c005210 0x7fcf4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 -- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf4c005210 msgr2=0x7fcf4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf4c005210 0x7fcf4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 -- 172.21.15.134:0/4176653141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf4c004800 msgr2=0x7fcf4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf4c004800 0x7fcf4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66a52640 1 -- 172.21.15.134:0/4176653141 reap_dead start 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 -- 172.21.15.134:0/4176653141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c003520 con 0x7fcf4c003a00 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.131+0000 7fcf66251640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf4c003e00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c110 tx=0x7fcf4c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=809baff2d80c430f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.132+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50007c40 con 0x7fcf4c003a00 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.132+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf50030070 con 0x7fcf4c003a00 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.132+0000 7fcf6524f640 1 -- 172.21.15.134:0/4176653141 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50005d10 con 0x7fcf4c003a00 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.132+0000 7fcf684b5640 1 -- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 msgr2=0x7fcf4c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.132+0000 7fcf684b5640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf4c003e00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c110 tx=0x7fcf4c005830 comp rx=0 tx=0).stop 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.133+0000 7fcf684b5640 1 -- 172.21.15.134:0/4176653141 shutdown_connections 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.133+0000 7fcf684b5640 1 --2- 172.21.15.134:0/4176653141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf4c003e00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.133+0000 7fcf684b5640 1 -- 172.21.15.134:0/4176653141 >> 172.21.15.134:0/4176653141 conn(0x7fcf6006c630 msgr2=0x7fcf6006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.133+0000 7fcf684b5640 1 -- 172.21.15.134:0/4176653141 shutdown_connections 2024-03-27T06:13:46.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.133+0000 7fcf684b5640 1 -- 172.21.15.134:0/4176653141 wait complete. 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.134+0000 7fcf684b5640 1 Processor -- start 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.134+0000 7fcf684b5640 1 -- start start 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 0x7fcf60150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 0x7fcf601535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6006b7d0 con 0x7fcf601de790 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcf600c8690 con 0x7fcf4c003a00 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fcf6006b950 con 0x7fcf60153130 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66a52640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 0x7fcf601535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36022/0 (socket says 172.21.15.134:36022) 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 -- 172.21.15.134:0/2652010623 learned_addr learned my addr 172.21.15.134:0/2652010623 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf65a50640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 0x7fcf60150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 -- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 msgr2=0x7fcf601535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 0x7fcf601535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 -- 172.21.15.134:0/2652010623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 msgr2=0x7fcf60150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 0x7fcf60150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 -- 172.21.15.134:0/2652010623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf601e2270 con 0x7fcf4c003a00 2024-03-27T06:13:46.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66a52640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 0x7fcf601535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf65a50640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 0x7fcf60150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf66251640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c230 tx=0x7fcf600c7ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50045070 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf500047e0 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50001b80 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf601debd0 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.136+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf601f15d0 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.137+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcf50004b90 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.137+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fcf50035d60 con 0x7fcf4c003a00 2024-03-27T06:13:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.137+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf60068950 con 0x7fcf4c003a00 2024-03-27T06:13:46.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.139+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcf5004e020 con 0x7fcf4c003a00 2024-03-27T06:13:46.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.213+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ed2023b3-8f13-4157-ba83-e8245614800c", "id": 5} v 0) -- 0x7fcf600b2e50 con 0x7fcf4c003a00 2024-03-27T06:13:46.208 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.217+0000 7f830ec49040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T06:13:46.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.218+0000 7fcf56ffd640 1 -- 172.21.15.134:0/2652010623 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed2023b3-8f13-4157-ba83-e8245614800c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fcf601e2270 con 0x7fcf4c003a00 2024-03-27T06:13:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:5 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 msgr2=0x7fcf601dfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c230 tx=0x7fcf600c7ce0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 shutdown_connections 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcf60153130 0x7fcf601535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf601de790 0x7fcf60150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 --2- 172.21.15.134:0/2652010623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf4c003a00 0x7fcf601dfb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 >> 172.21.15.134:0/2652010623 conn(0x7fcf6006c630 msgr2=0x7fcf60157140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 shutdown_connections 2024-03-27T06:13:46.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.219+0000 7fcf684b5640 1 -- 172.21.15.134:0/2652010623 wait complete. 2024-03-27T06:13:46.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new e8b1479f-defc-4abe-81b7-ed751bce3325 6 2024-03-27T06:13:46.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.329+0000 7f228bdf6640 1 Processor -- start 2024-03-27T06:13:46.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.329+0000 7f228bdf6640 1 -- start start 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f228bdf6640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2284152930 0x7f22840c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f228bdf6640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f22840ce7b0 0x7f22840c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f228bdf6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22840c3570 con 0x7f22840b0c60 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f228bdf6640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f22840c50f0 con 0x7f22840ce7b0 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f228bdf6640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f228406eb00 con 0x7f2284152930 2024-03-27T06:13:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2284152930 0x7f22840c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2284152930 0x7f22840c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36032/0 (socket says 172.21.15.134:36032) 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 -- 172.21.15.134:0/135848450 learned_addr learned my addr 172.21.15.134:0/135848450 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 -- 172.21.15.134:0/135848450 >> v2:172.21.15.132:3300/0 conn(0x7f22840ce7b0 msgr2=0x7f22840c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 --2- 172.21.15.134:0/135848450 >> v2:172.21.15.132:3300/0 conn(0x7f22840ce7b0 0x7f22840c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 -- 172.21.15.134:0/135848450 >> v1:172.21.15.132:6789/0 conn(0x7f22840b0c60 legacy=0x7f22840ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.330+0000 7f2289b92640 1 -- 172.21.15.134:0/135848450 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228406ede0 con 0x7f2284152930 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f2289b92640 1 --2- 172.21.15.134:0/135848450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2284152930 0x7f22840c8b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2284068200 tx=0x7f228002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=297bc8dc55915171 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f2288b90640 1 -- 172.21.15.134:0/135848450 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280034070 con 0x7f2284152930 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f2288b90640 1 -- 172.21.15.134:0/135848450 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2280030860 con 0x7f2284152930 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f2288b90640 1 -- 172.21.15.134:0/135848450 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280030c30 con 0x7f2284152930 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f228bdf6640 1 -- 172.21.15.134:0/135848450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2284152930 msgr2=0x7f22840c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.331+0000 7f228bdf6640 1 --2- 172.21.15.134:0/135848450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2284152930 0x7f22840c8b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2284068200 tx=0x7f228002b9a0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 -- 172.21.15.134:0/135848450 shutdown_connections 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 --2- 172.21.15.134:0/135848450 >> v2:172.21.15.132:3300/0 conn(0x7f22840ce7b0 0x7f22840c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 --2- 172.21.15.134:0/135848450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2284152930 0x7f22840c8b10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 -- 172.21.15.134:0/135848450 >> 172.21.15.134:0/135848450 conn(0x7f2284067280 msgr2=0x7f2284067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 -- 172.21.15.134:0/135848450 shutdown_connections 2024-03-27T06:13:46.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.332+0000 7f228bdf6640 1 -- 172.21.15.134:0/135848450 wait complete. 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 Processor -- start 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 -- start start 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2284152930 0x7f2284070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f22841e2b80 0x7f22841e3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f22840c3130 con 0x7f2284152930 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2284152d10 con 0x7f22840b0c60 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.333+0000 7f228bdf6640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f22840c32b0 con 0x7f22841e2b80 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36042/0 (socket says 172.21.15.134:36042) 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 -- 172.21.15.134:0/746660151 learned_addr learned my addr 172.21.15.134:0/746660151 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f228a393640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f22841e2b80 0x7f22841e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289391640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2284152930 0x7f2284070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 -- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f22841e2b80 msgr2=0x7f22841e3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f22841e2b80 0x7f22841e3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 -- 172.21.15.134:0/746660151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2284152930 msgr2=0x7f2284070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2284152930 0x7f2284070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 -- 172.21.15.134:0/746660151 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22840c6030 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f2289b92640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2280000c00 tx=0x7f22800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280034040 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2280039d90 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22840c83c0 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280043630 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22841f0a70 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f228004a060 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.334+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f228002f9a0 con 0x7f22840b0c60 2024-03-27T06:13:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.335+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2284068950 con 0x7f22840b0c60 2024-03-27T06:13:46.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.337+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f228003e070 con 0x7f22840b0c60 2024-03-27T06:13:46.350 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.360+0000 7f830ec49040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T06:13:46.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.410+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8b1479f-defc-4abe-81b7-ed751bce3325", "id": 6} v 0) -- 0x7f228406f580 con 0x7f22840b0c60 2024-03-27T06:13:46.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.415+0000 7f227a7fc640 1 -- 172.21.15.134:0/746660151 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8b1479f-defc-4abe-81b7-ed751bce3325", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f228002fc80 con 0x7f22840b0c60 2024-03-27T06:13:46.406 INFO:teuthology.orchestra.run.smithi134.stdout:6 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 msgr2=0x7f2284070ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2280000c00 tx=0x7f22800046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 shutdown_connections 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f22841e2b80 0x7f22841e3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2284152930 0x7f2284070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 --2- 172.21.15.134:0/746660151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22840b0c60 0x7f2284070ab0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 >> 172.21.15.134:0/746660151 conn(0x7f2284067280 msgr2=0x7f22840ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 shutdown_connections 2024-03-27T06:13:46.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.416+0000 7f228bdf6640 1 -- 172.21.15.134:0/746660151 wait complete. 2024-03-27T06:13:46.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 1ab5fc83-1a68-45ca-9b76-b0911127d9c6 7 2024-03-27T06:13:46.451 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:46.461+0000 7f2b70626040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T06:13:46.480 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.489+0000 7f830ec49040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T06:13:46.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.527+0000 7f54a37b6640 1 Processor -- start 2024-03-27T06:13:46.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.527+0000 7f54a37b6640 1 -- start start 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a37b6640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a37b6640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a37b6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549c0ccd30 con 0x7f549c154f80 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a37b6640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f549c155360 con 0x7f549c0d0740 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a37b6640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f549c0ccfd0 con 0x7f549c0cb2f0 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a0d51640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a1d53640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a0d51640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47964/0 (socket says 172.21.15.134:47964) 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a1d53640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36050/0 (socket says 172.21.15.134:36050) 2024-03-27T06:13:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.528+0000 7f54a0d51640 1 -- 172.21.15.134:0/53674251 learned_addr learned my addr 172.21.15.134:0/53674251 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a0d51640 1 -- 172.21.15.134:0/53674251 >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 msgr2=0x7f549c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a0d51640 1 --2- 172.21.15.134:0/53674251 >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a0d51640 1 -- 172.21.15.134:0/53674251 >> v1:172.21.15.134:6789/0 conn(0x7f549c154f80 legacy=0x7f549c15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a0d51640 1 -- 172.21.15.134:0/53674251 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c0cd2b0 con 0x7f549c0cb2f0 2024-03-27T06:13:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a1d53640 1 --2- 172.21.15.134:0/53674251 >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:46.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f54a0d51640 1 --2- 172.21.15.134:0/53674251 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f549802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69b1bc7880d3d8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.529+0000 7f5493fff640 1 -- 172.21.15.134:0/53674251 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498035020 con 0x7f549c0cb2f0 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f5493fff640 1 -- 172.21.15.134:0/53674251 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5498031640 con 0x7f549c0cb2f0 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f5493fff640 1 -- 172.21.15.134:0/53674251 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549802d040 con 0x7f549c0cb2f0 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 -- 172.21.15.134:0/53674251 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 msgr2=0x7f549c0d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 --2- 172.21.15.134:0/53674251 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f549802b8f0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 -- 172.21.15.134:0/53674251 shutdown_connections 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 --2- 172.21.15.134:0/53674251 >> v2:172.21.15.134:3300/0 conn(0x7f549c0d0740 0x7f549c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 --2- 172.21.15.134:0/53674251 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0cb2f0 0x7f549c0d0200 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 -- 172.21.15.134:0/53674251 >> 172.21.15.134:0/53674251 conn(0x7f549c06d9f0 msgr2=0x7f549c06de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 -- 172.21.15.134:0/53674251 shutdown_connections 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.530+0000 7f54a37b6640 1 -- 172.21.15.134:0/53674251 wait complete. 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.531+0000 7f54a37b6640 1 Processor -- start 2024-03-27T06:13:46.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 -- start start 2024-03-27T06:13:46.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 0x7f549c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c154f80 0x7f549c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f549c0cc0a0 con 0x7f549c154f80 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f549c0cd870 con 0x7f549c0cb2f0 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a37b6640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f549c0cc220 con 0x7f549c0d0740 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a1552640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 0x7f549c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a0d51640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a0d51640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.134:47970/0 (socket says 172.21.15.134:47970) 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.532+0000 7f54a0d51640 1 -- 172.21.15.134:0/304851582 learned_addr learned my addr 172.21.15.134:0/304851582 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-03-27T06:13:46.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 -- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 msgr2=0x7f549c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 0x7f549c15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 -- 172.21.15.134:0/304851582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c154f80 msgr2=0x7f549c157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c154f80 0x7f549c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 -- 172.21.15.134:0/304851582 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c15ac00 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a1552640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 0x7f549c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a0d51640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f549802b930 tx=0x7f54980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549803a9e0 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f549803ab80 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c15aee0 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54980443f0 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c1f0940 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5498038020 con 0x7f549c0d0740 2024-03-27T06:13:46.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.533+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f549804da50 con 0x7f549c0d0740 2024-03-27T06:13:46.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.534+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c068950 con 0x7f549c0d0740 2024-03-27T06:13:46.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.538+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f549803f070 con 0x7f549c0d0740 2024-03-27T06:13:46.530 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.540+0000 7f830ec49040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T06:13:46.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.610+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1ab5fc83-1a68-45ca-9b76-b0911127d9c6", "id": 7} v 0) -- 0x7f549c152710 con 0x7f549c0d0740 2024-03-27T06:13:46.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.620+0000 7f5491ffb640 1 -- 172.21.15.134:0/304851582 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ab5fc83-1a68-45ca-9b76-b0911127d9c6", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f549804d3f0 con 0x7f549c0d0740 2024-03-27T06:13:46.611 INFO:teuthology.orchestra.run.smithi134.stdout:7 2024-03-27T06:13:46.611 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:46.620+0000 7f2b70626040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.620+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 msgr2=0x7f549c1f1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.620+0000 7f54a37b6640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f549802b930 tx=0x7f54980046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 shutdown_connections 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c154f80 0x7f549c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f549c0d0740 0x7f549c1f1140 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 --2- 172.21.15.134:0/304851582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f549c0cb2f0 0x7f549c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 >> 172.21.15.134:0/304851582 conn(0x7f549c06d9f0 msgr2=0x7f549c153790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 shutdown_connections 2024-03-27T06:13:46.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-27T06:13:46.621+0000 7f54a37b6640 1 -- 172.21.15.134:0/304851582 wait complete. 2024-03-27T06:13:46.622 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T06:13:46.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-27T06:13:46.629 INFO:tasks.ceph.osd.0:Started 2024-03-27T06:13:46.629 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T06:13:46.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-27T06:13:46.636 INFO:tasks.ceph.osd.1:Started 2024-03-27T06:13:46.636 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T06:13:46.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-27T06:13:46.642 INFO:tasks.ceph.osd.2:Started 2024-03-27T06:13:46.642 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-27T06:13:46.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-27T06:13:46.647 INFO:tasks.ceph.osd.3:Started 2024-03-27T06:13:46.647 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-27T06:13:46.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-27T06:13:46.648 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.658+0000 7f830ec49040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T06:13:46.649 INFO:tasks.ceph.osd.4:Started 2024-03-27T06:13:46.649 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-27T06:13:46.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-27T06:13:46.651 INFO:tasks.ceph.osd.5:Started 2024-03-27T06:13:46.652 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-27T06:13:46.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-27T06:13:46.654 INFO:tasks.ceph.osd.6:Started 2024-03-27T06:13:46.654 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-27T06:13:46.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-27T06:13:46.656 INFO:tasks.ceph.osd.7:Started 2024-03-27T06:13:46.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:13:46.763 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.772+0000 7f830ec49040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T06:13:46.793 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:46.802+0000 7f2b70626040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T06:13:46.875 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:46.884+0000 7f2b70626040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T06:13:46.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.906+0000 7fdfaf200640 1 Processor -- start 2024-03-27T06:13:46.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.906+0000 7fdfaf200640 1 -- start start 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.907+0000 7fdfaf200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdfb00cd6a0 0x7fdfb006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfaf200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb015de10 con 0x7fdfb006be10 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfaf200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb015e0b0 con 0x7fdfb01549a0 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfaf200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb00cada0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfa7e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdfb00cd6a0 0x7fdfb006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfade00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdfb01549a0 0x7fdfb00cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55934/0 (socket says 172.21.15.132:55934) 2024-03-27T06:13:46.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfade00640 1 -- 172.21.15.132:0/3596159083 learned_addr learned my addr 172.21.15.132:0/3596159083 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:46.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894519556 0 0) 0x7fdfb015e0b0 con 0x7fdfb01549a0 2024-03-27T06:13:46.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf94003520 con 0x7fdfb01549a0 2024-03-27T06:13:46.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087438454 0 0) 0x7fdfb015de10 con 0x7fdfb006be10 2024-03-27T06:13:46.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfa7e00640 1 -- 172.21.15.132:0/3596159083 >> v1:172.21.15.134:6789/0 conn(0x7fdfb01549a0 legacy=0x7fdfb00cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfa7e00640 1 -- 172.21.15.132:0/3596159083 >> v1:172.21.15.132:6789/0 conn(0x7fdfb006be10 legacy=0x7fdfb015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.908+0000 7fdfa7e00640 1 -- 172.21.15.132:0/3596159083 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb015e390 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.909+0000 7fdfa7e00640 1 --2- 172.21.15.132:0/3596159083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb006b8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf980097a0 tx=0x7fdf9802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64beff4bdbdb1de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9802f810 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9802f9b0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfad400640 1 -- 172.21.15.132:0/3596159083 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9802fd80 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 -- 172.21.15.132:0/3596159083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00cd6a0 msgr2=0x7fdfb006b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 --2- 172.21.15.132:0/3596159083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb006b8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf980097a0 tx=0x7fdf9802b9f0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 -- 172.21.15.132:0/3596159083 shutdown_connections 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 --2- 172.21.15.132:0/3596159083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb006b8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 -- 172.21.15.132:0/3596159083 >> 172.21.15.132:0/3596159083 conn(0x7fdfb006d8d0 msgr2=0x7fdfb006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.910+0000 7fdfaf200640 1 -- 172.21.15.132:0/3596159083 shutdown_connections 2024-03-27T06:13:46.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.911+0000 7fdfaf200640 1 -- 172.21.15.132:0/3596159083 wait complete. 2024-03-27T06:13:46.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.912+0000 7fdfaf200640 1 Processor -- start 2024-03-27T06:13:46.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.912+0000 7fdfaf200640 1 -- start start 2024-03-27T06:13:46.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 0x7fdfb01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 0x7fdfb01468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb015e950 con 0x7fdfb01549a0 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb00cdad0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfaf200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb015ec30 con 0x7fdfb006be10 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfae800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 0x7fdfb01468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfade00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 0x7fdfb01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfa7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfa7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56572/0 (socket says 172.21.15.132:56572) 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfa7e00640 1 -- 172.21.15.132:0/2712597620 learned_addr learned my addr 172.21.15.132:0/2712597620 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.913+0000 7fdfade00640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 0x7fdfb01591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:46948/0 (socket says 172.21.15.132:46948) 2024-03-27T06:13:46.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 -- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 msgr2=0x7fdfb01591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 0x7fdfb01591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 -- 172.21.15.132:0/2712597620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 msgr2=0x7fdfb01468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 0x7fdfb01468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 -- 172.21.15.132:0/2712597620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb01589e0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfae800640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 0x7fdfb01468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa7e00640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdf980098c0 tx=0x7fdf98004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98047070 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf98037810 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb0158cc0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.914+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98037be0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.915+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb01525f0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.915+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf9804a070 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.915+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fdf98042ac0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.916+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb006b850 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.918+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdf98033030 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.920 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.930+0000 7f830ec49040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T06:13:46.962 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:46.972+0000 7f830ec49040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T06:13:46.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.984+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdfb00686c0 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.984+0000 7fdfa6000640 1 -- 172.21.15.132:0/2712597620 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fdf9802f910 con 0x7fdfb00cd6a0 2024-03-27T06:13:46.976 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:46.976 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":10,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:13:46.613151+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:13:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 msgr2=0x7fdfb014f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdf980098c0 tx=0x7fdf98004730 comp rx=0 tx=0).stop 2024-03-27T06:13:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 shutdown_connections 2024-03-27T06:13:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb01549a0 0x7fdfb01468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfb00cd6a0 0x7fdfb014f0b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 --2- 172.21.15.132:0/2712597620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdfb006be10 0x7fdfb01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:46.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.986+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 >> 172.21.15.132:0/2712597620 conn(0x7fdfb006d8d0 msgr2=0x7fdfb00c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:46.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.987+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 shutdown_connections 2024-03-27T06:13:46.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:46.987+0000 7fdfaf200640 1 -- 172.21.15.132:0/2712597620 wait complete. 2024-03-27T06:13:46.993 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T06:13:46.994 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T06:13:47.012 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:47.021+0000 7f2b70626040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T06:13:47.119 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:47.128+0000 7f830ec49040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T06:13:47.124 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:47.134+0000 7f2b70626040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T06:13:47.253 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-03-27T06:13:47.262+0000 7fb665211640 -1 Falling back to public interface 2024-03-27T06:13:47.261 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-03-27T06:13:47.270+0000 7fe674985640 -1 Falling back to public interface 2024-03-27T06:13:47.266 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-03-27T06:13:47.275+0000 7fa154222640 -1 Falling back to public interface 2024-03-27T06:13:47.273 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-03-27T06:13:47.282+0000 7f94f4b0d640 -1 Falling back to public interface 2024-03-27T06:13:47.303 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:47.311+0000 7f2b70626040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T06:13:47.326 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-27T06:13:47.336+0000 7fbaa036f640 -1 Falling back to public interface 2024-03-27T06:13:47.352 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-27T06:13:47.362+0000 7f4a7e138640 -1 Falling back to public interface 2024-03-27T06:13:47.353 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:47.363+0000 7f2b70626040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T06:13:47.355 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-27T06:13:47.365+0000 7ffa5900d640 -1 Falling back to public interface 2024-03-27T06:13:47.361 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-27T06:13:47.370+0000 7f9ea38b3640 -1 Falling back to public interface 2024-03-27T06:13:47.536 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:47.545+0000 7f2b70626040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T06:13:47.798 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T06:13:47.907 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:47.916+0000 7f830ec49040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T06:13:47.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.963+0000 7f8a52800640 1 Processor -- start 2024-03-27T06:13:47.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.964+0000 7f8a52800640 1 -- start start 2024-03-27T06:13:47.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a52800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8a4c072c60 0x7f8a4c15e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:47.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a52800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c073040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:47.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a52800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c0cb140 con 0x7f8a4c15ada0 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a52800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4c15a910 con 0x7f8a4c072c60 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a52800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0cb420 con 0x7f8a4c0c5270 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a4b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8a4c072c60 0x7f8a4c15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a4be00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8a4c15ada0 0x7f8a4c15b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:53190/0 (socket says 172.21.15.132:53190) 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a4be00640 1 -- 172.21.15.132:0/369976416 learned_addr learned my addr 172.21.15.132:0/369976416 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a43e00640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c073040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.965+0000 7f8a4b400640 1 -- 172.21.15.132:0/369976416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 msgr2=0x7f8a4c073040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.966+0000 7f8a4b400640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c073040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.966+0000 7f8a4b400640 1 -- 172.21.15.132:0/369976416 >> v1:172.21.15.132:6789/0 conn(0x7f8a4c15ada0 legacy=0x7f8a4c15b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.966+0000 7f8a4b400640 1 -- 172.21.15.132:0/369976416 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0cc310 con 0x7f8a4c072c60 2024-03-27T06:13:47.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.966+0000 7f8a4aa00640 1 -- 172.21.15.132:0/369976416 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506688884 0 0) 0x7f8a4c0cb140 con 0x7f8a4c15ada0 2024-03-27T06:13:47.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.967+0000 7f8a4b400640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c15e810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097a0 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a3a880ee89bf8b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:47.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.967+0000 7f8a4aa00640 1 -- 172.21.15.132:0/369976416 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c035020 con 0x7f8a4c072c60 2024-03-27T06:13:47.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.967+0000 7f8a4aa00640 1 -- 172.21.15.132:0/369976416 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c031830 con 0x7f8a4c072c60 2024-03-27T06:13:47.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.967+0000 7f8a4aa00640 1 -- 172.21.15.132:0/369976416 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c031c00 con 0x7f8a4c072c60 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 -- 172.21.15.132:0/369976416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 msgr2=0x7f8a4c15e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c15e810 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097a0 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 -- 172.21.15.132:0/369976416 shutdown_connections 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c073040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 --2- 172.21.15.132:0/369976416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c15e810 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 -- 172.21.15.132:0/369976416 >> 172.21.15.132:0/369976416 conn(0x7f8a4c0d0140 msgr2=0x7f8a4c0d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 -- 172.21.15.132:0/369976416 shutdown_connections 2024-03-27T06:13:47.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.968+0000 7f8a52800640 1 -- 172.21.15.132:0/369976416 wait complete. 2024-03-27T06:13:47.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.970+0000 7f8a52800640 1 Processor -- start 2024-03-27T06:13:47.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.970+0000 7f8a52800640 1 -- start start 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a4c15ada0 0x7f8a4c1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c154ea0 con 0x7f8a4c072c60 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0cc8d0 con 0x7f8a4c15ada0 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a52800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0c58e0 con 0x7f8a4c0c5270 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a4b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:47.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:46958/0 (socket says 172.21.15.132:46958) 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 -- 172.21.15.132:0/1036400773 learned_addr learned my addr 172.21.15.132:0/1036400773 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a4be00640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a4c15ada0 0x7f8a4c1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 -- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a4c15ada0 msgr2=0x7f8a4c1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a4c15ada0 0x7f8a4c1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 -- 172.21.15.132:0/1036400773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 msgr2=0x7f8a4c153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.972+0000 7f8a43e00640 1 -- 172.21.15.132:0/1036400773 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0b5db0 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43e00640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e9e0 tx=0x7f8a3400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34017070 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3400c9d0 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c157b90 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400cdc0 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c158150 con 0x7f8a4c0c5270 2024-03-27T06:13:47.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8a3401a070 con 0x7f8a4c0c5270 2024-03-27T06:13:47.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.973+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8a3401cc60 con 0x7f8a4c0c5270 2024-03-27T06:13:47.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.974+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c075b00 con 0x7f8a4c0c5270 2024-03-27T06:13:47.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:47.977+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8a3401c590 con 0x7f8a4c0c5270 2024-03-27T06:13:48.035 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.044+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a4c14ef20 con 0x7f8a4c0c5270 2024-03-27T06:13:48.035 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.045+0000 7f8a43400640 1 -- 172.21.15.132:0/1036400773 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8a3401c770 con 0x7f8a4c0c5270 2024-03-27T06:13:48.036 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-27T06:13:48.036 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:13:46.613151+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 msgr2=0x7f8a4c1f3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e9e0 tx=0x7f8a3400eea0 comp rx=0 tx=0).stop 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 shutdown_connections 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a4c15ada0 0x7f8a4c1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a4c0c5270 0x7f8a4c1f3ee0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 --2- 172.21.15.132:0/1036400773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c072c60 0x7f8a4c153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 >> 172.21.15.132:0/1036400773 conn(0x7f8a4c0d0140 msgr2=0x7f8a4c0d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 shutdown_connections 2024-03-27T06:13:48.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:48.046+0000 7f8a52800640 1 -- 172.21.15.132:0/1036400773 wait complete. 2024-03-27T06:13:48.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T06:13:48.188 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:48.197+0000 7f830ec49040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T06:13:48.238 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:48.247+0000 7f830ec49040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T06:13:48.353 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:48.362+0000 7f830ec49040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T06:13:48.579 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:48.588+0000 7f2b70626040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T06:13:48.895 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:48.904+0000 7f2b70626040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T06:13:48.955 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:48.964+0000 7f2b70626040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T06:13:49.089 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-27T06:13:49.098+0000 7f2b70626040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T06:13:49.392 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:49.401+0000 7f82af84c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-27T06:13:49.392 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:13:49.401+0000 7f82af84c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-27T06:13:50.731 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-03-27T06:13:50.740+0000 7fa154222640 -1 osd.4 0 log_to_monitors true 2024-03-27T06:13:50.758 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-03-27T06:13:50.767+0000 7fb665211640 -1 osd.6 0 log_to_monitors true 2024-03-27T06:13:50.791 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-03-27T06:13:50.800+0000 7f94f4b0d640 -1 osd.7 0 log_to_monitors true 2024-03-27T06:13:50.793 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-03-27T06:13:50.802+0000 7fe674985640 -1 osd.5 0 log_to_monitors true 2024-03-27T06:13:50.912 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-27T06:13:50.922+0000 7ffa5900d640 -1 osd.1 0 log_to_monitors true 2024-03-27T06:13:50.941 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-27T06:13:50.950+0000 7f9ea38b3640 -1 osd.2 0 log_to_monitors true 2024-03-27T06:13:51.125 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-27T06:13:51.134+0000 7fbaa036f640 -1 osd.0 0 log_to_monitors true 2024-03-27T06:13:51.192 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-27T06:13:51.201+0000 7f4a7e138640 -1 osd.3 0 log_to_monitors true 2024-03-27T06:13:53.396 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-03-27T06:13:53.405+0000 7fa14fc1a640 -1 osd.4 0 waiting for initial osdmap 2024-03-27T06:13:53.410 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-03-27T06:13:53.419+0000 7fa147a3a640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.417 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-03-27T06:13:53.426+0000 7fe67091b640 -1 osd.5 0 waiting for initial osdmap 2024-03-27T06:13:53.419 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-03-27T06:13:53.428+0000 7fb6611a7640 -1 osd.6 0 waiting for initial osdmap 2024-03-27T06:13:53.426 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-03-27T06:13:53.436+0000 7fe66873b640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.427 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-03-27T06:13:53.436+0000 7fb658fc7640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.460 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-03-27T06:13:53.469+0000 7f94f0aa3640 -1 osd.7 0 waiting for initial osdmap 2024-03-27T06:13:53.472 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-03-27T06:13:53.482+0000 7f94e88c3640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.829 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-27T06:13:53.838+0000 7f9e9f200640 -1 osd.2 0 waiting for initial osdmap 2024-03-27T06:13:53.838 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-27T06:13:53.847+0000 7f9e94800640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.841 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-27T06:13:53.850+0000 7f4a79000640 -1 osd.3 0 waiting for initial osdmap 2024-03-27T06:13:53.851 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-27T06:13:53.860+0000 7f4a6f000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.862 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-27T06:13:53.872+0000 7ffa53e00640 -1 osd.1 0 waiting for initial osdmap 2024-03-27T06:13:53.866 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-27T06:13:53.876+0000 7ffa49e00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:53.887 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-27T06:13:53.897+0000 7fba9b200640 -1 osd.0 0 waiting for initial osdmap 2024-03-27T06:13:53.894 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-27T06:13:53.903+0000 7fba91200640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T06:13:54.856 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T06:13:55.036 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.045+0000 7f621b600640 1 Processor -- start 2024-03-27T06:13:55.036 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.045+0000 7f621b600640 1 -- start start 2024-03-27T06:13:55.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.046+0000 7f621b600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f62141511f0 0x7f62140c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f621b600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62140c1f90 con 0x7f62140732a0 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f621b600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62141515d0 con 0x7f62140c0f50 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f621b600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f62140c7ec0 con 0x7f62141511f0 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6218c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f62141511f0 0x7f62140c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6213e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f62140c0f50 0x7f62140b17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60282/0 (socket says 172.21.15.132:60282) 2024-03-27T06:13:55.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6213e00640 1 -- 172.21.15.132:0/2093730674 learned_addr learned my addr 172.21.15.132:0/2093730674 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:55.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6218c00640 1 -- 172.21.15.132:0/2093730674 >> v1:172.21.15.134:6789/0 conn(0x7f62140c0f50 legacy=0x7f62140b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6218c00640 1 -- 172.21.15.132:0/2093730674 >> v1:172.21.15.132:6789/0 conn(0x7f62140732a0 legacy=0x7f6214161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6218c00640 1 -- 172.21.15.132:0/2093730674 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62140c16f0 con 0x7f62141511f0 2024-03-27T06:13:55.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.047+0000 7f6213400640 1 -- 172.21.15.132:0/2093730674 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105346153 0 0) 0x7f62141515d0 con 0x7f62140c0f50 2024-03-27T06:13:55.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.048+0000 7f6218c00640 1 --2- 172.21.15.132:0/2093730674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62141511f0 0x7f62140c84c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0097a0 tx=0x7f61fc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee90e8851d0cd90 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f6213400640 1 -- 172.21.15.132:0/2093730674 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc034070 con 0x7f62141511f0 2024-03-27T06:13:55.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f6213400640 1 -- 172.21.15.132:0/2093730674 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc030610 con 0x7f62141511f0 2024-03-27T06:13:55.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f6213400640 1 -- 172.21.15.132:0/2093730674 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0309e0 con 0x7f62141511f0 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 -- 172.21.15.132:0/2093730674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62141511f0 msgr2=0x7f62140c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 --2- 172.21.15.132:0/2093730674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62141511f0 0x7f62140c84c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0097a0 tx=0x7f61fc02b960 comp rx=0 tx=0).stop 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 -- 172.21.15.132:0/2093730674 shutdown_connections 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 --2- 172.21.15.132:0/2093730674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62141511f0 0x7f62140c84c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 -- 172.21.15.132:0/2093730674 >> 172.21.15.132:0/2093730674 conn(0x7f62140cbf30 msgr2=0x7f62140cc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 -- 172.21.15.132:0/2093730674 shutdown_connections 2024-03-27T06:13:55.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.049+0000 7f621b600640 1 -- 172.21.15.132:0/2093730674 wait complete. 2024-03-27T06:13:55.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.051+0000 7f621b600640 1 Processor -- start 2024-03-27T06:13:55.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.051+0000 7f621b600640 1 -- start start 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f62140c0f50 0x7f621414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62141511f0 0x7f6214143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f62140c7ab0 con 0x7f62141511f0 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f62140c1cb0 con 0x7f62140732a0 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f621b600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f62140c7c30 con 0x7f62140c0f50 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6218c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6213e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f62140c0f50 0x7f621414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6219600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62141511f0 0x7f6214143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6218c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39620/0 (socket says 172.21.15.132:39620) 2024-03-27T06:13:55.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6218c00640 1 -- 172.21.15.132:0/2160762473 learned_addr learned my addr 172.21.15.132:0/2160762473 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6218c00640 1 -- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f62140c0f50 msgr2=0x7f621414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.052+0000 7f6218c00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f62140c0f50 0x7f621414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6218c00640 1 -- 172.21.15.132:0/2160762473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62141511f0 msgr2=0x7f6214143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6218c00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62141511f0 0x7f6214143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6218c00640 1 -- 172.21.15.132:0/2160762473 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621415b1e0 con 0x7f62140732a0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6218c00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0098c0 tx=0x7f61fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc034040 con 0x7f62140732a0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc03acc0 con 0x7f62140732a0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621414d780 con 0x7f62140732a0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc044630 con 0x7f62140732a0 2024-03-27T06:13:55.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.053+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6214142d40 con 0x7f62140732a0 2024-03-27T06:13:55.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.054+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6214075b00 con 0x7f62140732a0 2024-03-27T06:13:55.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.055+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f61fc037070 con 0x7f62140732a0 2024-03-27T06:13:55.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.056+0000 7f6210c00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 0x7f61f006e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.056+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f61fc0ad1b0 con 0x7f62140732a0 2024-03-27T06:13:55.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.056+0000 7f6213e00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 0x7f61f006e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.057+0000 7f6213e00640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 0x7f61f006e990 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6204001920 tx=0x7f62040018b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.061+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f61fc03f070 con 0x7f62140732a0 2024-03-27T06:13:55.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.248+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62140c82a0 con 0x7f62140732a0 2024-03-27T06:13:55.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.249+0000 7f6210c00640 1 -- 172.21.15.132:0/2160762473 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10945 (secure 0 0 0) 0x7f61fc07b7b0 con 0x7f62140732a0 2024-03-27T06:13:55.240 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-27T06:13:55.241 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":13,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:13:54.436616+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:13:55.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 msgr2=0x7f61f006e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 0x7f61f006e990 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6204001920 tx=0x7f62040018b0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 msgr2=0x7f621414c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0098c0 tx=0x7f61fc0046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 shutdown_connections 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62141511f0 0x7f6214143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f62140c0f50 0x7f621414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f61f006c4d0 0x7f61f006e990 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 --2- 172.21.15.132:0/2160762473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62140732a0 0x7f621414c9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 >> 172.21.15.132:0/2160762473 conn(0x7f62140cbf30 msgr2=0x7f621414ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:55.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 shutdown_connections 2024-03-27T06:13:55.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-27T06:13:55.252+0000 7f621b600640 1 -- 172.21.15.132:0/2160762473 wait complete. 2024-03-27T06:13:55.256 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-27T06:13:55.256 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-27T06:13:55.256 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-27T06:13:55.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-27T06:13:55.263 INFO:tasks.ceph.mds.a:Started 2024-03-27T06:13:55.263 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-27T06:13:55.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-27T06:13:55.270 INFO:tasks.ceph.mds.c:Started 2024-03-27T06:13:55.270 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-27T06:13:55.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-03-27T06:13:55.276 INFO:tasks.ceph.mds.e:Started 2024-03-27T06:13:55.276 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-27T06:13:55.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-27T06:13:55.279 INFO:tasks.ceph.mds.b:Started 2024-03-27T06:13:55.279 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-27T06:13:55.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-27T06:13:55.281 INFO:tasks.ceph.mds.d:Started 2024-03-27T06:13:55.281 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-27T06:13:55.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-27T06:13:55.283 INFO:tasks.ceph.mds.f:Started 2024-03-27T06:13:55.284 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-27T06:13:55.284 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:13:55.364 INFO:tasks.ceph.mds.d.smithi134.stdout:starting mds.d at 2024-03-27T06:13:55.365 INFO:tasks.ceph.mds.b.smithi134.stdout:starting mds.b at 2024-03-27T06:13:55.365 INFO:tasks.ceph.mds.f.smithi134.stdout:starting mds.f at 2024-03-27T06:13:55.365 INFO:tasks.ceph.mds.b.smithi134.stderr:2024-03-27T06:13:55.375+0000 7fdfb38c4640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-03-27T06:13:55.410 INFO:tasks.ceph.mds.a.smithi132.stdout:starting mds.a at 2024-03-27T06:13:55.417 INFO:tasks.ceph.mds.e.smithi132.stdout:starting mds.e at 2024-03-27T06:13:55.420 INFO:tasks.ceph.mds.c.smithi132.stdout:starting mds.c at 2024-03-27T06:13:55.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.473+0000 7ff85c000640 1 Processor -- start 2024-03-27T06:13:55.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.473+0000 7ff85c000640 1 -- start start 2024-03-27T06:13:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85c000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff8541590a0 0x7ff854071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85c000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85406b630 con 0x7ff854152950 2024-03-27T06:13:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85c000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85406b8d0 con 0x7ff8540716b0 2024-03-27T06:13:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85c000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7ff854152d30 con 0x7ff8541590a0 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff858c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff8541590a0 0x7ff854071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85a000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff8540716b0 0x7ff85406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60310/0 (socket says 172.21.15.132:60310) 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff859600640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ff854152950 0x7ff854073550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42348/0 (socket says 172.21.15.132:42348) 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff85a000640 1 -- 172.21.15.132:0/1158241812 learned_addr learned my addr 172.21.15.132:0/1158241812 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433170864 0 0) 0x7ff85406b8d0 con 0x7ff8540716b0 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff83c003520 con 0x7ff8540716b0 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240824994 0 0) 0x7ff85406b630 con 0x7ff854152950 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff85406b8d0 con 0x7ff854152950 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff858c00640 1 -- 172.21.15.132:0/1158241812 >> v1:172.21.15.134:6789/0 conn(0x7ff8540716b0 legacy=0x7ff85406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff858c00640 1 -- 172.21.15.132:0/1158241812 >> v1:172.21.15.132:6789/0 conn(0x7ff854152950 legacy=0x7ff854073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.475+0000 7ff858c00640 1 -- 172.21.15.132:0/1158241812 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85406bbb0 con 0x7ff8541590a0 2024-03-27T06:13:55.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.476+0000 7ff858c00640 1 --2- 172.21.15.132:0/1158241812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8541590a0 0x7ff854071170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8400097a0 tx=0x7ff84002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed44d8d3d0deeecc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840035020 con 0x7ff8541590a0 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff840031830 con 0x7ff8541590a0 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff853e00640 1 -- 172.21.15.132:0/1158241812 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840031c00 con 0x7ff8541590a0 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 -- 172.21.15.132:0/1158241812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8541590a0 msgr2=0x7ff854071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 --2- 172.21.15.132:0/1158241812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8541590a0 0x7ff854071170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8400097a0 tx=0x7ff84002b8d0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 -- 172.21.15.132:0/1158241812 shutdown_connections 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 --2- 172.21.15.132:0/1158241812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8541590a0 0x7ff854071170 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 -- 172.21.15.132:0/1158241812 >> 172.21.15.132:0/1158241812 conn(0x7ff8540cc060 msgr2=0x7ff8540cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 -- 172.21.15.132:0/1158241812 shutdown_connections 2024-03-27T06:13:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.477+0000 7ff85c000640 1 -- 172.21.15.132:0/1158241812 wait complete. 2024-03-27T06:13:55.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.479+0000 7ff85c000640 1 Processor -- start 2024-03-27T06:13:55.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.479+0000 7ff85c000640 1 -- start start 2024-03-27T06:13:55.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540716b0 0x7ff85414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 0x7ff854147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff85406c450 con 0x7ff854152950 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff85406c170 con 0x7ff8540716b0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85c000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff8541594c0 con 0x7ff8541590a0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff858c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff858c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32836/0 (socket says 172.21.15.132:32836) 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff85a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 0x7ff854147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff858c00640 1 -- 172.21.15.132:0/4203795007 learned_addr learned my addr 172.21.15.132:0/4203795007 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.480+0000 7ff859600640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540716b0 0x7ff85414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 -- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540716b0 msgr2=0x7ff85414d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540716b0 0x7ff85414d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 -- 172.21.15.132:0/4203795007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 msgr2=0x7ff85415bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 -- 172.21.15.132:0/4203795007 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8540c4b00 con 0x7ff8541590a0 2024-03-27T06:13:55.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff858c00640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:55.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85a000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 0x7ff854147680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00cab0 tx=0x7ff84c00cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c017070 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84c00ac30 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c012600 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8540c4de0 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.481+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8540c53a0 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.482+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff854068840 con 0x7ff8541590a0 2024-03-27T06:13:55.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.483+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7ff84c01a070 con 0x7ff8541590a0 2024-03-27T06:13:55.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.484+0000 7ff851600640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 0x7ff83406e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.484+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7ff84c08adc0 con 0x7ff8541590a0 2024-03-27T06:13:55.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.484+0000 7ff859600640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 0x7ff83406e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.484+0000 7ff859600640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 0x7ff83406e990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff848007ac0 tx=0x7ff848007a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.488+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff84c00ae20 con 0x7ff8541590a0 2024-03-27T06:13:55.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.677+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8540c6b30 con 0x7ff8541590a0 2024-03-27T06:13:55.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.678+0000 7ff851600640 1 -- 172.21.15.132:0/4203795007 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+11129 (secure 0 0 0) 0x7ff84c0592a0 con 0x7ff8541590a0 2024-03-27T06:13:55.669 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:55.669 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:13:55.454260+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:13:55.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.680+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 msgr2=0x7ff83406e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.680+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 0x7ff83406e990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff848007ac0 tx=0x7ff848007a10 comp rx=0 tx=0).stop 2024-03-27T06:13:55.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.680+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 msgr2=0x7ff854147680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.680+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 0x7ff854147680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00cab0 tx=0x7ff84c00cf70 comp rx=0 tx=0).stop 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 shutdown_connections 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8541590a0 0x7ff854147680 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff854152950 0x7ff85415bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff83406c4d0 0x7ff83406e990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 --2- 172.21.15.132:0/4203795007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540716b0 0x7ff85414d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 >> 172.21.15.132:0/4203795007 conn(0x7ff8540cc060 msgr2=0x7ff854074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 shutdown_connections 2024-03-27T06:13:55.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.681+0000 7ff85c000640 1 -- 172.21.15.132:0/4203795007 wait complete. 2024-03-27T06:13:55.687 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-27T06:13:55.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:13:55.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.901+0000 7fefd7200640 1 Processor -- start 2024-03-27T06:13:55.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.901+0000 7fefd7200640 1 -- start start 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fefd81549c0 0x7fefd815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fefd806c7a0 0x7fefd806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd806bf80 0x7fefd8072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fefd8071d80 con 0x7fefd81549c0 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fefd8071220 con 0x7fefd806c7a0 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd7200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fefd806d020 con 0x7fefd806bf80 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd5e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fefd81549c0 0x7fefd815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd5e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fefd81549c0 0x7fefd815b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32846/0 (socket says 172.21.15.132:32846) 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 -- 172.21.15.132:0/2404116147 learned_addr learned my addr 172.21.15.132:0/2404116147 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.902+0000 7fefd6800640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd806bf80 0x7fefd8072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefcfe00640 1 --2- 172.21.15.132:0/2404116147 >> v2:172.21.15.134:3300/0 conn(0x7fefd806c7a0 0x7fefd806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 -- 172.21.15.132:0/2404116147 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd806bf80 msgr2=0x7fefd8072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd806bf80 0x7fefd8072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 -- 172.21.15.132:0/2404116147 >> v2:172.21.15.134:3300/0 conn(0x7fefd806c7a0 msgr2=0x7fefd806ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2404116147 >> v2:172.21.15.134:3300/0 conn(0x7fefd806c7a0 0x7fefd806ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.903+0000 7fefd5e00640 1 -- 172.21.15.132:0/2404116147 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd806d300 con 0x7fefd81549c0 2024-03-27T06:13:55.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.904+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd81549c0 0x7fefd815b1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefd8073730 tx=0x7fefc802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c129054ad393fda8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.904+0000 7fefd5400640 1 -- 172.21.15.132:0/2404116147 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8034070 con 0x7fefd81549c0 2024-03-27T06:13:55.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.904+0000 7fefd5400640 1 -- 172.21.15.132:0/2404116147 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc8030860 con 0x7fefd81549c0 2024-03-27T06:13:55.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.904+0000 7fefd5400640 1 -- 172.21.15.132:0/2404116147 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8030c30 con 0x7fefd81549c0 2024-03-27T06:13:55.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 -- 172.21.15.132:0/2404116147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd81549c0 msgr2=0x7fefd815b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd81549c0 0x7fefd815b1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefd8073730 tx=0x7fefc802b9a0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 -- 172.21.15.132:0/2404116147 shutdown_connections 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd806bf80 0x7fefd8072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 --2- 172.21.15.132:0/2404116147 >> v2:172.21.15.134:3300/0 conn(0x7fefd806c7a0 0x7fefd806ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 --2- 172.21.15.132:0/2404116147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd81549c0 0x7fefd815b1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 -- 172.21.15.132:0/2404116147 >> 172.21.15.132:0/2404116147 conn(0x7fefd80cc7a0 msgr2=0x7fefd80c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 -- 172.21.15.132:0/2404116147 shutdown_connections 2024-03-27T06:13:55.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.905+0000 7fefd7200640 1 -- 172.21.15.132:0/2404116147 wait complete. 2024-03-27T06:13:55.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.907+0000 7fefd7200640 1 Processor -- start 2024-03-27T06:13:55.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.907+0000 7fefd7200640 1 -- start start 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.908+0000 7fefd7200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 0x7fefd8159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.908+0000 7fefd7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 0x7fefd80c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.908+0000 7fefd7200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd81549c0 0x7fefd815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd7200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8071a10 con 0x7fefd806c7a0 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd7200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8071730 con 0x7fefd806bf80 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd7200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fefd806cbc0 con 0x7fefd81549c0 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefcfe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 0x7fefd80c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 0x7fefd8159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefcfe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 0x7fefd80c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32856/0 (socket says 172.21.15.132:32856) 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd6800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd81549c0 0x7fefd815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefcfe00640 1 -- 172.21.15.132:0/2125244409 learned_addr learned my addr 172.21.15.132:0/2125244409 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 -- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd81549c0 msgr2=0x7fefd815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd81549c0 0x7fefd815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 -- 172.21.15.132:0/2125244409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 msgr2=0x7fefd80c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 0x7fefd80c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 -- 172.21.15.132:0/2125244409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd81e4da0 con 0x7fefd806bf80 2024-03-27T06:13:55.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.909+0000 7fefd5e00640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 0x7fefd8159750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefc8033000 tx=0x7fefc80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.910+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8034040 con 0x7fefd806bf80 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.910+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc8030e40 con 0x7fefd806bf80 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.910+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc80433f0 con 0x7fefd806bf80 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.910+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd81e5040 con 0x7fefd806bf80 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.910+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd81e5600 con 0x7fefd806bf80 2024-03-27T06:13:55.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.911+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd80bf520 con 0x7fefd806bf80 2024-03-27T06:13:55.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.912+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7fefc8037070 con 0x7fefd806bf80 2024-03-27T06:13:55.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.913+0000 7fefce000640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 0x7fefb406e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:55.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.913+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7fefc80ad130 con 0x7fefd806bf80 2024-03-27T06:13:55.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.913+0000 7fefcfe00640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 0x7fefb406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:55.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.913+0000 7fefcfe00640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 0x7fefb406e960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fefd80c4d10 tx=0x7fefc0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:55.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:55.917+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fefd80bf520 con 0x7fefd806bf80 2024-03-27T06:13:56.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.158+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefd80686c0 con 0x7fefd806bf80 2024-03-27T06:13:56.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.159+0000 7fefce000640 1 -- 172.21.15.132:0/2125244409 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7fefc803e070 con 0x7fefd806bf80 2024-03-27T06:13:56.151 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:56.151 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-27T06:13:56.151 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 2 2024-03-27T06:13:56.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.161+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 msgr2=0x7fefb406e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.161+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 0x7fefb406e960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fefd80c4d10 tx=0x7fefc0009190 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 msgr2=0x7fefd8159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 0x7fefd8159750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefc8033000 tx=0x7fefc80046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 shutdown_connections 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fefd81549c0 0x7fefd815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefd806c7a0 0x7fefd80c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fefb406c4a0 0x7fefb406e960 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 --2- 172.21.15.132:0/2125244409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefd806bf80 0x7fefd8159750 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 >> 172.21.15.132:0/2125244409 conn(0x7fefd80cc7a0 msgr2=0x7fefd80cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 shutdown_connections 2024-03-27T06:13:56.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:56.162+0000 7fefd7200640 1 -- 172.21.15.132:0/2125244409 wait complete. 2024-03-27T06:13:58.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:13:58.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.396+0000 7f1459800640 1 Processor -- start 2024-03-27T06:13:58.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.396+0000 7f1459800640 1 -- start start 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1459800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 0x7f14540cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1459800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f145415adb0 0x7f145415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1459800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145415de10 con 0x7f14540cb870 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1459800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f14540cada0 con 0x7f145415adb0 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1459800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f145415b720 con 0x7f14541549a0 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f144b400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f14540cb870 0x7f145406b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42362/0 (socket says 172.21.15.132:42362) 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1453e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 0x7f14540cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f144b400640 1 -- 172.21.15.132:0/3208507518 learned_addr learned my addr 172.21.15.132:0/3208507518 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:58.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.398+0000 7f1458e00640 1 --2- 172.21.15.132:0/3208507518 >> v2:172.21.15.134:3300/0 conn(0x7f145415adb0 0x7f145415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.399+0000 7f1453e00640 1 -- 172.21.15.132:0/3208507518 >> v2:172.21.15.134:3300/0 conn(0x7f145415adb0 msgr2=0x7f145415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:58.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.399+0000 7f1453e00640 1 --2- 172.21.15.132:0/3208507518 >> v2:172.21.15.134:3300/0 conn(0x7f145415adb0 0x7f145415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.399+0000 7f1453e00640 1 -- 172.21.15.132:0/3208507518 >> v1:172.21.15.132:6789/0 conn(0x7f14540cb870 legacy=0x7f145406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.399+0000 7f1453e00640 1 -- 172.21.15.132:0/3208507518 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145415e0b0 con 0x7f14541549a0 2024-03-27T06:13:58.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.399+0000 7f1453400640 1 -- 172.21.15.132:0/3208507518 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110832472 0 0) 0x7f145415de10 con 0x7f14540cb870 2024-03-27T06:13:58.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.400+0000 7f1453e00640 1 --2- 172.21.15.132:0/3208507518 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 0x7f14540cd6a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f144402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd6cf7dc4458eedb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.400+0000 7f1453400640 1 -- 172.21.15.132:0/3208507518 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039070 con 0x7f14541549a0 2024-03-27T06:13:58.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.400+0000 7f1453400640 1 -- 172.21.15.132:0/3208507518 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144402cce0 con 0x7f14541549a0 2024-03-27T06:13:58.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.400+0000 7f1453400640 1 -- 172.21.15.132:0/3208507518 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444034660 con 0x7f14541549a0 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 -- 172.21.15.132:0/3208507518 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 msgr2=0x7f14540cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 --2- 172.21.15.132:0/3208507518 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 0x7f14540cd6a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f144402bf30 comp rx=0 tx=0).stop 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 -- 172.21.15.132:0/3208507518 shutdown_connections 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 --2- 172.21.15.132:0/3208507518 >> v2:172.21.15.134:3300/0 conn(0x7f145415adb0 0x7f145415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 --2- 172.21.15.132:0/3208507518 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14541549a0 0x7f14540cd6a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 -- 172.21.15.132:0/3208507518 >> 172.21.15.132:0/3208507518 conn(0x7f145406d8d0 msgr2=0x7f145406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 -- 172.21.15.132:0/3208507518 shutdown_connections 2024-03-27T06:13:58.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.401+0000 7f1459800640 1 -- 172.21.15.132:0/3208507518 wait complete. 2024-03-27T06:13:58.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.403+0000 7f1459800640 1 Processor -- start 2024-03-27T06:13:58.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.403+0000 7f1459800640 1 -- start start 2024-03-27T06:13:58.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 0x7f14540ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145415adb0 0x7f1454153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f145415e950 con 0x7f145415adb0 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f145415e670 con 0x7f14541549a0 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1459800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f145415ec30 con 0x7f14540cb870 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f144b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 0x7f14540ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50184/0 (socket says 172.21.15.132:50184) 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1458e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145415adb0 0x7f1454153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 -- 172.21.15.132:0/1271133861 learned_addr learned my addr 172.21.15.132:0/1271133861 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 -- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 msgr2=0x7f14540ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 0x7f14540ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 -- 172.21.15.132:0/1271133861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145415adb0 msgr2=0x7f1454153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145415adb0 0x7f1454153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.404+0000 7f1453e00640 1 -- 172.21.15.132:0/1271133861 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145415de00 con 0x7f14540cb870 2024-03-27T06:13:58.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f144b400640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 0x7f14540ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:58.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1453e00640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f1444004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039050 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144403a040 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14541f4670 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444009450 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.405+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14540ce380 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.407+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14540686c0 con 0x7f14540cb870 2024-03-27T06:13:58.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.407+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f1444047020 con 0x7f14540cb870 2024-03-27T06:13:58.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.408+0000 7f1450c00640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 0x7f143006e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.408+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f14440ad3b0 con 0x7f14540cb870 2024-03-27T06:13:58.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.408+0000 7f144b400640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 0x7f143006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.409+0000 7f144b400640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 0x7f143006e960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f145415d0b0 tx=0x7f143c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.413+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f144404e020 con 0x7f14540cb870 2024-03-27T06:13:58.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.659+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14540cafd0 con 0x7f14540cb870 2024-03-27T06:13:58.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.660+0000 7f1450c00640 1 -- 172.21.15.132:0/1271133861 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5527 (secure 0 0 0) 0x7f145415de00 con 0x7f14540cb870 2024-03-27T06:13:58.652 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:58.652 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4273,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-27T06:13:58.652 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 3 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 msgr2=0x7f143006e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 0x7f143006e960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f145415d0b0 tx=0x7f143c007d50 comp rx=0 tx=0).stop 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 msgr2=0x7f14541f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f1444004a40 comp rx=0 tx=0).stop 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 shutdown_connections 2024-03-27T06:13:58.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145415adb0 0x7f1454153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143006c4a0 0x7f143006e960 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14541549a0 0x7f14540ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 --2- 172.21.15.132:0/1271133861 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540cb870 0x7f14541f4130 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 >> 172.21.15.132:0/1271133861 conn(0x7f145406d8d0 msgr2=0x7f1454149e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 shutdown_connections 2024-03-27T06:13:58.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.663+0000 7f1459800640 1 -- 172.21.15.132:0/1271133861 wait complete. 2024-03-27T06:13:58.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:13:58.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.904+0000 7f685b400640 1 Processor -- start 2024-03-27T06:13:58.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.905+0000 7f685b400640 1 -- start start 2024-03-27T06:13:58.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f685b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f685406c8a0 0x7f68540c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f685b400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 0x7f68540c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f685b400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68540cdad0 con 0x7f68540c77f0 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f685b400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f68540cc210 con 0x7f685406c8a0 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f685b400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f685415b690 con 0x7f685406cc80 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f6858a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f685406c8a0 0x7f68540c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f6859400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f68540c77f0 0x7f6854154890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42366/0 (socket says 172.21.15.132:42366) 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f6859400640 1 -- 172.21.15.132:0/2091956079 learned_addr learned my addr 172.21.15.132:0/2091956079 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.906+0000 7f6853e00640 1 --2- 172.21.15.132:0/2091956079 >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 0x7f68540c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.907+0000 7f6858a00640 1 -- 172.21.15.132:0/2091956079 >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 msgr2=0x7f68540c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.907+0000 7f6858a00640 1 --2- 172.21.15.132:0/2091956079 >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 0x7f68540c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.907+0000 7f6858a00640 1 -- 172.21.15.132:0/2091956079 >> v1:172.21.15.132:6789/0 conn(0x7f68540c77f0 legacy=0x7f6854154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.907+0000 7f6858a00640 1 -- 172.21.15.132:0/2091956079 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685415b970 con 0x7f685406c8a0 2024-03-27T06:13:58.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.907+0000 7f6853e00640 1 --2- 172.21.15.132:0/2091956079 >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 0x7f68540c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.908+0000 7f6858a00640 1 --2- 172.21.15.132:0/2091956079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c8320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f68541587a0 tx=0x7f684802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9708fa5840b60d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.908+0000 7f6853400640 1 -- 172.21.15.132:0/2091956079 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68480315a0 con 0x7f685406c8a0 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.908+0000 7f6853400640 1 -- 172.21.15.132:0/2091956079 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6848037070 con 0x7f685406c8a0 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.908+0000 7f6853400640 1 -- 172.21.15.132:0/2091956079 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848031960 con 0x7f685406c8a0 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 -- 172.21.15.132:0/2091956079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 msgr2=0x7f68540c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 --2- 172.21.15.132:0/2091956079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c8320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f68541587a0 tx=0x7f684802b840 comp rx=0 tx=0).stop 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 -- 172.21.15.132:0/2091956079 shutdown_connections 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 --2- 172.21.15.132:0/2091956079 >> v2:172.21.15.134:3300/0 conn(0x7f685406cc80 0x7f68540c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 --2- 172.21.15.132:0/2091956079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c8320 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 -- 172.21.15.132:0/2091956079 >> 172.21.15.132:0/2091956079 conn(0x7f68540ccbe0 msgr2=0x7f68540ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:58.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 -- 172.21.15.132:0/2091956079 shutdown_connections 2024-03-27T06:13:58.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.909+0000 7f685b400640 1 -- 172.21.15.132:0/2091956079 wait complete. 2024-03-27T06:13:58.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.911+0000 7f685b400640 1 Processor -- start 2024-03-27T06:13:58.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.911+0000 7f685b400640 1 -- start start 2024-03-27T06:13:58.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f685406cc80 0x7f68541f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68540c77f0 0x7f685415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f685406d440 con 0x7f685406c8a0 2024-03-27T06:13:58.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f685406d2c0 con 0x7f68540c77f0 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f685b400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f68540cd700 con 0x7f685406cc80 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32892/0 (socket says 172.21.15.132:32892) 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 -- 172.21.15.132:0/3343678114 learned_addr learned my addr 172.21.15.132:0/3343678114 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6853e00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f685406cc80 0x7f68541f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6859400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68540c77f0 0x7f685415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 -- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f685406cc80 msgr2=0x7f68541f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f685406cc80 0x7f68541f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 -- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68540c77f0 msgr2=0x7f685415e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68540c77f0 0x7f685415e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:58.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.912+0000 7f6858a00640 1 -- 172.21.15.132:0/3343678114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68541f0b90 con 0x7f685406c8a0 2024-03-27T06:13:58.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f6858a00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6848000c00 tx=0x7f68480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848032630 con 0x7f685406c8a0 2024-03-27T06:13:58.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68480327d0 con 0x7f685406c8a0 2024-03-27T06:13:58.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848032ba0 con 0x7f685406c8a0 2024-03-27T06:13:58.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68541f0e50 con 0x7f685406c8a0 2024-03-27T06:13:58.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.913+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68540c5010 con 0x7f685406c8a0 2024-03-27T06:13:58.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.915+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68540c51e0 con 0x7f685406c8a0 2024-03-27T06:13:58.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.916+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f684804c020 con 0x7f685406c8a0 2024-03-27T06:13:58.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.917+0000 7f6850c00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 0x7f683406e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:58.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.917+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f684802f020 con 0x7f685406c8a0 2024-03-27T06:13:58.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.918+0000 7f6853e00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 0x7f683406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:58.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.918+0000 7f6853e00640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 0x7f683406e960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6840009560 tx=0x7f6840009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:58.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:58.928+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f68540c51e0 con 0x7f685406c8a0 2024-03-27T06:13:59.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.135+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68540c6910 con 0x7f685406c8a0 2024-03-27T06:13:59.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.136+0000 7f6850c00640 1 -- 172.21.15.132:0/3343678114 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12933 (secure 0 0 0) 0x7f6848040070 con 0x7f685406c8a0 2024-03-27T06:13:59.128 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:59.128 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":16,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:13:58.392859+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:13:59.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 msgr2=0x7f683406e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 0x7f683406e960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6840009560 tx=0x7f6840009190 comp rx=0 tx=0).stop 2024-03-27T06:13:59.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 msgr2=0x7f68540c3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6848000c00 tx=0x7f68480046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 shutdown_connections 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f683406c4a0 0x7f683406e960 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68540c77f0 0x7f685415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f685406cc80 0x7f68541f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 --2- 172.21.15.132:0/3343678114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685406c8a0 0x7f68540c3f90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 >> 172.21.15.132:0/3343678114 conn(0x7f68540ccbe0 msgr2=0x7f6854158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 shutdown_connections 2024-03-27T06:13:59.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.139+0000 7f685b400640 1 -- 172.21.15.132:0/3343678114 wait complete. 2024-03-27T06:13:59.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:13:59.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:13:59.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.346+0000 7f9f74400640 1 Processor -- start 2024-03-27T06:13:59.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.346+0000 7f9f74400640 1 -- start start 2024-03-27T06:13:59.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.347+0000 7f9f74400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 0x7f9f6c0c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.347+0000 7f9f74400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 0x7f9f6c072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.347+0000 7f9f74400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f6c06c530 con 0x7f9f6c151bf0 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.347+0000 7f9f74400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f9f6c073150 con 0x7f9f6c158360 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.347+0000 7f9f74400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6c06c7d0 con 0x7f9f6c151810 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 0x7f9f6c0c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71000640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f9f6c151bf0 0x7f9f6c072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42374/0 (socket says 172.21.15.132:42374) 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71000640 1 -- 172.21.15.132:0/1051089527 learned_addr learned my addr 172.21.15.132:0/1051089527 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f72400640 1 --2- 172.21.15.132:0/1051089527 >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 0x7f9f6c072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71a00640 1 -- 172.21.15.132:0/1051089527 >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 msgr2=0x7f9f6c072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71a00640 1 --2- 172.21.15.132:0/1051089527 >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 0x7f9f6c072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71a00640 1 -- 172.21.15.132:0/1051089527 >> v1:172.21.15.132:6789/0 conn(0x7f9f6c151bf0 legacy=0x7f9f6c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f71a00640 1 -- 172.21.15.132:0/1051089527 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6c06cab0 con 0x7f9f6c151810 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f72400640 1 --2- 172.21.15.132:0/1051089527 >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 0x7f9f6c072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.348+0000 7f9f6be00640 1 -- 172.21.15.132:0/1051089527 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291919963 0 0) 0x7f9f6c06c530 con 0x7f9f6c151bf0 2024-03-27T06:13:59.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f71a00640 1 --2- 172.21.15.132:0/1051089527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 0x7f9f6c0c1d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f600097a0 tx=0x7f9f6002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=896568249b9d2087 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f6be00640 1 -- 172.21.15.132:0/1051089527 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60034070 con 0x7f9f6c151810 2024-03-27T06:13:59.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f6be00640 1 -- 172.21.15.132:0/1051089527 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f60030800 con 0x7f9f6c151810 2024-03-27T06:13:59.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f6be00640 1 -- 172.21.15.132:0/1051089527 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60030bd0 con 0x7f9f6c151810 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 -- 172.21.15.132:0/1051089527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 msgr2=0x7f9f6c0c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 --2- 172.21.15.132:0/1051089527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 0x7f9f6c0c1d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f600097a0 tx=0x7f9f6002b940 comp rx=0 tx=0).stop 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 -- 172.21.15.132:0/1051089527 shutdown_connections 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 --2- 172.21.15.132:0/1051089527 >> v2:172.21.15.134:3300/0 conn(0x7f9f6c158360 0x7f9f6c072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 --2- 172.21.15.132:0/1051089527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c151810 0x7f9f6c0c1d80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.350+0000 7f9f74400640 1 -- 172.21.15.132:0/1051089527 >> 172.21.15.132:0/1051089527 conn(0x7f9f6c0c89a0 msgr2=0x7f9f6c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.351+0000 7f9f74400640 1 -- 172.21.15.132:0/1051089527 shutdown_connections 2024-03-27T06:13:59.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.351+0000 7f9f74400640 1 -- 172.21.15.132:0/1051089527 wait complete. 2024-03-27T06:13:59.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.352+0000 7f9f74400640 1 Processor -- start 2024-03-27T06:13:59.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.352+0000 7f9f74400640 1 -- start start 2024-03-27T06:13:59.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 0x7f9f6c14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f6c14f2a0 0x7f9f6c1f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c157e40 con 0x7f9f6c151810 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c157b60 con 0x7f9f6c14f2a0 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.353+0000 7f9f74400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6c0c22c0 con 0x7f9f6c158360 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 0x7f9f6c14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 0x7f9f6c14ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32906/0 (socket says 172.21.15.132:32906) 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f72400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f6c14f2a0 0x7f9f6c1f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50222/0 (socket says 172.21.15.132:50222) 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71a00640 1 -- 172.21.15.132:0/2418465097 learned_addr learned my addr 172.21.15.132:0/2418465097 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 -- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f6c14f2a0 msgr2=0x7f9f6c1f07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f6c14f2a0 0x7f9f6c1f07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 -- 172.21.15.132:0/2418465097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 msgr2=0x7f9f6c14ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 0x7f9f6c14ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 -- 172.21.15.132:0/2418465097 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6c158b30 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f71000640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800c450 tx=0x7f9f5800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f580074e0 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.354+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f580047e0 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.355+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58004bd0 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.355+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6c14f990 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.355+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6c14ff50 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.356+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6c068950 con 0x7f9f6c158360 2024-03-27T06:13:59.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.357+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f9f58011070 con 0x7f9f6c158360 2024-03-27T06:13:59.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.357+0000 7f9f69600640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 0x7f9f4c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.358+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9f5808ad70 con 0x7f9f6c158360 2024-03-27T06:13:59.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.358+0000 7f9f71a00640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 0x7f9f4c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.358+0000 7f9f71a00640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 0x7f9f4c06e960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f600097a0 tx=0x7f9f60008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.362+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9f58059720 con 0x7f9f6c158360 2024-03-27T06:13:59.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.566+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9f6c0711c0 con 0x7f9f6c158360 2024-03-27T06:13:59.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.567+0000 7f9f69600640 1 -- 172.21.15.132:0/2418465097 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f9f58059020 con 0x7f9f6c158360 2024-03-27T06:13:59.559 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 msgr2=0x7f9f4c06e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 0x7f9f4c06e960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f600097a0 tx=0x7f9f60008030 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 msgr2=0x7f9f6c0c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800c450 tx=0x7f9f5800c850 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 shutdown_connections 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f4c06c4a0 0x7f9f4c06e960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.569+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f6c14f2a0 0x7f9f6c1f07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.570+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f6c158360 0x7f9f6c0c18f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.570+0000 7f9f74400640 1 --2- 172.21.15.132:0/2418465097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f6c151810 0x7f9f6c14ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.570+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 >> 172.21.15.132:0/2418465097 conn(0x7f9f6c0c89a0 msgr2=0x7f9f6c06d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.570+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 shutdown_connections 2024-03-27T06:13:59.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.570+0000 7f9f74400640 1 -- 172.21.15.132:0/2418465097 wait complete. 2024-03-27T06:13:59.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-27T06:13:59.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.778+0000 7f858b800640 1 Processor -- start 2024-03-27T06:13:59.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.778+0000 7f858b800640 1 -- start start 2024-03-27T06:13:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.779+0000 7f858b800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8584158f10 0x7f858415f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.779+0000 7f858b800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.779+0000 7f858b800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f858406b520 0x7f858406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f858b800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f858415fc20 con 0x7f858406b520 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f858b800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f85840c1120 con 0x7f8584158f10 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f858b800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f85840c13e0 con 0x7f858406d3e0 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8589800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f858406b520 0x7f858406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8588e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8584158f10 0x7f858415f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50232/0 (socket says 172.21.15.132:50232) 2024-03-27T06:13:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 -- 172.21.15.132:0/2306681146 learned_addr learned my addr 172.21.15.132:0/2306681146 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 -- 172.21.15.132:0/2306681146 >> v2:172.21.15.134:3300/0 conn(0x7f8584158f10 msgr2=0x7f858415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 --2- 172.21.15.132:0/2306681146 >> v2:172.21.15.134:3300/0 conn(0x7f8584158f10 0x7f858415f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 -- 172.21.15.132:0/2306681146 >> v2:172.21.15.132:3300/0 conn(0x7f858406b520 msgr2=0x7f858406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 --2- 172.21.15.132:0/2306681146 >> v2:172.21.15.132:3300/0 conn(0x7f858406b520 0x7f858406b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.780+0000 7f8583e00640 1 -- 172.21.15.132:0/2306681146 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85840c16c0 con 0x7f858406d3e0 2024-03-27T06:13:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.781+0000 7f8583e00640 1 --2- 172.21.15.132:0/2306681146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f858406c7b0 tx=0x7f857002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0bb36f66340c93a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f8583400640 1 -- 172.21.15.132:0/2306681146 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570035020 con 0x7f858406d3e0 2024-03-27T06:13:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f8583400640 1 -- 172.21.15.132:0/2306681146 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8570031870 con 0x7f858406d3e0 2024-03-27T06:13:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f8583400640 1 -- 172.21.15.132:0/2306681146 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570031c40 con 0x7f858406d3e0 2024-03-27T06:13:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 -- 172.21.15.132:0/2306681146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 msgr2=0x7f858406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 --2- 172.21.15.132:0/2306681146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f858406c7b0 tx=0x7f857002b910 comp rx=0 tx=0).stop 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 -- 172.21.15.132:0/2306681146 shutdown_connections 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 --2- 172.21.15.132:0/2306681146 >> v2:172.21.15.132:3300/0 conn(0x7f858406b520 0x7f858406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 --2- 172.21.15.132:0/2306681146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f858406d800 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 --2- 172.21.15.132:0/2306681146 >> v2:172.21.15.134:3300/0 conn(0x7f8584158f10 0x7f858415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 -- 172.21.15.132:0/2306681146 >> 172.21.15.132:0/2306681146 conn(0x7f85840cc060 msgr2=0x7f85840cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 -- 172.21.15.132:0/2306681146 shutdown_connections 2024-03-27T06:13:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.782+0000 7f858b800640 1 -- 172.21.15.132:0/2306681146 wait complete. 2024-03-27T06:13:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.784+0000 7f858b800640 1 Processor -- start 2024-03-27T06:13:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.784+0000 7f858b800640 1 -- start start 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 0x7f85841eb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8584158f10 0x7f858414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85840c1f60 con 0x7f8584158f10 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f85840c1c80 con 0x7f858406b520 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f858b800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f858406cec0 con 0x7f858406d3e0 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8583e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8588e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 0x7f85841eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8583e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50240/0 (socket says 172.21.15.132:50240) 2024-03-27T06:13:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8589800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8584158f10 0x7f858414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8588e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 0x7f85841eb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39762/0 (socket says 172.21.15.132:39762) 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.785+0000 7f8583e00640 1 -- 172.21.15.132:0/3974967416 learned_addr learned my addr 172.21.15.132:0/3974967416 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 -- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 msgr2=0x7f85841eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 0x7f85841eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 -- 172.21.15.132:0/3974967416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8584158f10 msgr2=0x7f858414e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8584158f10 0x7f858414e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 -- 172.21.15.132:0/3974967416 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858415f120 con 0x7f858406d3e0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8583e00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f857002b950 tx=0x7f85700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570040070 con 0x7f858406d3e0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8570031e10 con 0x7f858406d3e0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570035040 con 0x7f858406d3e0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85841ebeb0 con 0x7f858406d3e0 2024-03-27T06:13:59.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.786+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85841eabb0 con 0x7f858406d3e0 2024-03-27T06:13:59.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.788+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8584068840 con 0x7f858406d3e0 2024-03-27T06:13:59.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.788+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f857004b060 con 0x7f858406d3e0 2024-03-27T06:13:59.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.789+0000 7f8580c00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 0x7f856406e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:13:59.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.789+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f85700acd10 con 0x7f858406d3e0 2024-03-27T06:13:59.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.789+0000 7f8588e00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 0x7f856406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:13:59.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.790+0000 7f8588e00640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 0x7f856406e960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8578009a20 tx=0x7f8578009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:13:59.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.794+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f857007b1a0 con 0x7f858406d3e0 2024-03-27T06:13:59.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.996+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f858415d2a0 con 0x7f858406d3e0 2024-03-27T06:13:59.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:13:59.997+0000 7f8580c00640 1 -- 172.21.15.132:0/3974967416 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f857007f4a0 con 0x7f858406d3e0 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout:[ 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout: "poolnum": 1, 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout: "poolname": ".mgr" 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-27T06:13:59.989 INFO:teuthology.orchestra.run.smithi132.stdout:] 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 msgr2=0x7f856406e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 0x7f856406e960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8578009a20 tx=0x7f8578009190 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 msgr2=0x7f8584071f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f857002b950 tx=0x7f85700046c0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 shutdown_connections 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8584158f10 0x7f858414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f858406d3e0 0x7f8584071f10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f856406c4a0 0x7f856406e960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 --2- 172.21.15.132:0/3974967416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f858406b520 0x7f85841eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 >> 172.21.15.132:0/3974967416 conn(0x7f85840cc060 msgr2=0x7f85840cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:13:59.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 shutdown_connections 2024-03-27T06:13:59.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.000+0000 7f858b800640 1 -- 172.21.15.132:0/3974967416 wait complete. 2024-03-27T06:14:00.008 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-27T06:14:00.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-27T06:14:00.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.219+0000 7f9d44c00640 1 Processor -- start 2024-03-27T06:14:00.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.219+0000 7f9d44c00640 1 -- start start 2024-03-27T06:14:00.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d44c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 0x7f9d400c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d44c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9d4006c700 0x7f9d400c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d44c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d400719b0 con 0x7f9d40152900 2024-03-27T06:14:00.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d44c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f9d400c83b0 con 0x7f9d4006c700 2024-03-27T06:14:00.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d44c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9d40071c90 con 0x7f9d40072a50 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d3de00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9d40152900 0x7f9d40159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60316/0 (socket says 172.21.15.132:60316) 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d3de00640 1 -- 172.21.15.132:0/1201151210 learned_addr learned my addr 172.21.15.132:0/1201151210 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d3e800640 1 --2- 172.21.15.132:0/1201151210 >> v2:172.21.15.132:3300/0 conn(0x7f9d4006c700 0x7f9d400c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.221+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1201151210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 0x7f9d400c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d3d400640 1 -- 172.21.15.132:0/1201151210 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372091398 0 0) 0x7f9d400719b0 con 0x7f9d40152900 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d3d400640 1 -- 172.21.15.132:0/1201151210 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d20003520 con 0x7f9d40152900 2024-03-27T06:14:00.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d37e00640 1 -- 172.21.15.132:0/1201151210 >> v2:172.21.15.132:3300/0 conn(0x7f9d4006c700 msgr2=0x7f9d400c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:00.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1201151210 >> v2:172.21.15.132:3300/0 conn(0x7f9d4006c700 0x7f9d400c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:00.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d37e00640 1 -- 172.21.15.132:0/1201151210 >> v1:172.21.15.134:6789/0 conn(0x7f9d40152900 legacy=0x7f9d40159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:00.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.222+0000 7f9d37e00640 1 -- 172.21.15.132:0/1201151210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d40071f70 con 0x7f9d40072a50 2024-03-27T06:14:00.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.223+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1201151210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 0x7f9d400c7b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d280097a0 tx=0x7f9d2802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dcfa7dc80ed0d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:00.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.273+0000 7f9d3d400640 1 -- 172.21.15.132:0/1201151210 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28034070 con 0x7f9d40072a50 2024-03-27T06:14:00.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.273+0000 7f9d3d400640 1 -- 172.21.15.132:0/1201151210 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d28030800 con 0x7f9d40072a50 2024-03-27T06:14:00.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.273+0000 7f9d3d400640 1 -- 172.21.15.132:0/1201151210 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28030bb0 con 0x7f9d40072a50 2024-03-27T06:14:00.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 -- 172.21.15.132:0/1201151210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 msgr2=0x7f9d400c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:00.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 --2- 172.21.15.132:0/1201151210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 0x7f9d400c7b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d280097a0 tx=0x7f9d2802b940 comp rx=0 tx=0).stop 2024-03-27T06:14:00.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 -- 172.21.15.132:0/1201151210 shutdown_connections 2024-03-27T06:14:00.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 --2- 172.21.15.132:0/1201151210 >> v2:172.21.15.132:3300/0 conn(0x7f9d4006c700 0x7f9d400c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:00.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 --2- 172.21.15.132:0/1201151210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d40072a50 0x7f9d400c7b70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:00.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.274+0000 7f9d44c00640 1 -- 172.21.15.132:0/1201151210 >> 172.21.15.132:0/1201151210 conn(0x7f9d400c8a80 msgr2=0x7f9d400c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:00.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.275+0000 7f9d44c00640 1 -- 172.21.15.132:0/1201151210 shutdown_connections 2024-03-27T06:14:00.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.275+0000 7f9d44c00640 1 -- 172.21.15.132:0/1201151210 wait complete. 2024-03-27T06:14:00.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.277+0000 7f9d44c00640 1 Processor -- start 2024-03-27T06:14:00.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.278+0000 7f9d44c00640 1 -- start start 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d40072a50 0x7f9d40155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40152900 0x7f9d40151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d400c80b0 con 0x7f9d40152900 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9d40072530 con 0x7f9d40072a50 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d44c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4006cb30 con 0x7f9d4006c700 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d3de00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d3de00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50254/0 (socket says 172.21.15.132:50254) 2024-03-27T06:14:00.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d3de00640 1 -- 172.21.15.132:0/1080983320 learned_addr learned my addr 172.21.15.132:0/1080983320 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:00.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d40072a50 0x7f9d40155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.279+0000 7f9d3e800640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40152900 0x7f9d40151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 -- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d40072a50 msgr2=0x7f9d40155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d40072a50 0x7f9d40155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 -- 172.21.15.132:0/1080983320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40152900 msgr2=0x7f9d40151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40152900 0x7f9d40151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 -- 172.21.15.132:0/1080983320 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d401ed760 con 0x7f9d4006c700 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d3de00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c460 tx=0x7f9d3000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30017070 con 0x7f9d4006c700 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d30005ce0 con 0x7f9d4006c700 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d44c00640 1 -- 172.21.15.132:0/1080983320 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4014f990 con 0x7f9d4006c700 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.280+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30007270 con 0x7f9d4006c700 2024-03-27T06:14:00.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.281+0000 7f9d44c00640 1 -- 172.21.15.132:0/1080983320 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4014ff50 con 0x7f9d4006c700 2024-03-27T06:14:00.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.283+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d401563a0 con 0x7f9d4006c700 2024-03-27T06:14:00.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.284+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 4) ==== 90800+0+0 (secure 0 0 0) 0x7f9d3001a070 con 0x7f9d4006c700 2024-03-27T06:14:00.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.285+0000 7f9d36000640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 0x7f9d1c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:00.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.285+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9d3008a520 con 0x7f9d4006c700 2024-03-27T06:14:00.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.285+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 0x7f9d1c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:00.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.286+0000 7f9d37e00640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 0x7f9d1c06e960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d28036000 tx=0x7f9d2803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:00.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.290+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9d401563a0 con 0x7f9d4006c700 2024-03-27T06:14:00.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.492+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9d40068840 con 0x7f9d4006c700 2024-03-27T06:14:00.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:00.786+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9d3008a210 con 0x7f9d4006c700 2024-03-27T06:14:01.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.489+0000 7f9d36000640 1 -- 172.21.15.132:0/1080983320 <== mon.2 v2:172.21.15.134:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f9d30078b10 con 0x7f9d4006c700 2024-03-27T06:14:01.481 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'cephfs_metadata' created 2024-03-27T06:14:01.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.496+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 msgr2=0x7f9d1c06e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:01.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.496+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 0x7f9d1c06e960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d28036000 tx=0x7f9d2803b040 comp rx=0 tx=0).stop 2024-03-27T06:14:01.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.496+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 msgr2=0x7f9d401ed220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:01.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.496+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c460 tx=0x7f9d3000c920 comp rx=0 tx=0).stop 2024-03-27T06:14:01.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 shutdown_connections 2024-03-27T06:14:01.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40152900 0x7f9d40151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d1c06c4a0 0x7f9d1c06e960 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d40072a50 0x7f9d40155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 --2- 172.21.15.132:0/1080983320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d4006c700 0x7f9d401ed220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.497+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 >> 172.21.15.132:0/1080983320 conn(0x7f9d400c8a80 msgr2=0x7f9d400c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.498+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 shutdown_connections 2024-03-27T06:14:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.498+0000 7f9d2f400640 1 -- 172.21.15.132:0/1080983320 wait complete. 2024-03-27T06:14:01.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-27T06:14:01.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.717+0000 7fc99b400640 1 Processor -- start 2024-03-27T06:14:01.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.717+0000 7fc99b400640 1 -- start start 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc9940c9570 0x7fc99415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 0x7fc994157b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fc9940c1230 con 0x7fc99415e230 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fc9940c6ec0 con 0x7fc9940c9570 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc99b400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc9940c7180 con 0x7fc99415eb50 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc998a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc9940c9570 0x7fc99415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc999400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 0x7fc994157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc9940c9570 0x7fc99415e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39784/0 (socket says 172.21.15.132:39784) 2024-03-27T06:14:01.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc999400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 0x7fc994157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50264/0 (socket says 172.21.15.132:50264) 2024-03-27T06:14:01.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc998a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32946/0 (socket says 172.21.15.132:32946) 2024-03-27T06:14:01.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 -- 172.21.15.132:0/3650616717 learned_addr learned my addr 172.21.15.132:0/3650616717 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 -- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 msgr2=0x7fc994157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 --2- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 0x7fc994157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 -- 172.21.15.132:0/3650616717 >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 msgr2=0x7fc9940c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.719+0000 7fc993e00640 1 --2- 172.21.15.132:0/3650616717 >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.720+0000 7fc993e00640 1 -- 172.21.15.132:0/3650616717 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940c7460 con 0x7fc9940c9570 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.720+0000 7fc998a00640 1 --2- 172.21.15.132:0/3650616717 >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.721+0000 7fc993e00640 1 --2- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9940c9570 0x7fc99415e610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc99404fff0 tx=0x7fc98002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2393d46d6e6fd64c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.721+0000 7fc993400640 1 -- 172.21.15.132:0/3650616717 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980034070 con 0x7fc9940c9570 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.721+0000 7fc993400640 1 -- 172.21.15.132:0/3650616717 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc980030860 con 0x7fc9940c9570 2024-03-27T06:14:01.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.721+0000 7fc993400640 1 -- 172.21.15.132:0/3650616717 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980030c30 con 0x7fc9940c9570 2024-03-27T06:14:01.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 -- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9940c9570 msgr2=0x7fc99415e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:01.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 --2- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9940c9570 0x7fc99415e610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc99404fff0 tx=0x7fc98002b9a0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 -- 172.21.15.132:0/3650616717 shutdown_connections 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 --2- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc99415eb50 0x7fc994157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 --2- 172.21.15.132:0/3650616717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9940c9570 0x7fc99415e610 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 --2- 172.21.15.132:0/3650616717 >> v2:172.21.15.132:3300/0 conn(0x7fc99415e230 0x7fc9940c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 -- 172.21.15.132:0/3650616717 >> 172.21.15.132:0/3650616717 conn(0x7fc9940c89a0 msgr2=0x7fc9940c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 -- 172.21.15.132:0/3650616717 shutdown_connections 2024-03-27T06:14:01.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.722+0000 7fc99b400640 1 -- 172.21.15.132:0/3650616717 wait complete. 2024-03-27T06:14:01.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.724+0000 7fc99b400640 1 Processor -- start 2024-03-27T06:14:01.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.724+0000 7fc99b400640 1 -- start start 2024-03-27T06:14:01.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc9940c9570 0x7fc9941f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 0x7fc994153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc994158090 con 0x7fc99415e230 2024-03-27T06:14:01.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940c0ec0 con 0x7fc99415eb50 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc99b400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc994158210 con 0x7fc9940c9570 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32962/0 (socket says 172.21.15.132:32962) 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 -- 172.21.15.132:0/1271515554 learned_addr learned my addr 172.21.15.132:0/1271515554 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc998a00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc9940c9570 0x7fc9941f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc999400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 0x7fc994153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 -- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc9940c9570 msgr2=0x7fc9941f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc9940c9570 0x7fc9941f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 -- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 msgr2=0x7fc994153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 0x7fc994153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc993e00640 1 -- 172.21.15.132:0/1271515554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99415dd10 con 0x7fc99415e230 2024-03-27T06:14:01.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.725+0000 7fc999400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 0x7fc994153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:01.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc993e00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc980000c00 tx=0x7fc980004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:01.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980048070 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc980030660 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980034040 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99415bbb0 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.726+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99415c170 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.727+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc994068950 con 0x7fc99415e230 2024-03-27T06:14:01.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.728+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc98004b070 con 0x7fc99415e230 2024-03-27T06:14:01.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.729+0000 7fc990c00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 0x7fc97406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:01.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.729+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fc9800ad410 con 0x7fc99415e230 2024-03-27T06:14:01.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.730+0000 7fc998a00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 0x7fc97406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:01.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.730+0000 7fc998a00640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 0x7fc97406e9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc988009560 tx=0x7fc988009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:01.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.735+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc980030830 con 0x7fc99415e230 2024-03-27T06:14:01.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:01.935+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc994153770 con 0x7fc99415e230 2024-03-27T06:14:02.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.498+0000 7fc990c00640 1 -- 172.21.15.132:0/1271515554 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fc98007b5b0 con 0x7fc99415e230 2024-03-27T06:14:02.491 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'cephfs_data' created 2024-03-27T06:14:02.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.503+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 msgr2=0x7fc97406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.503+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 0x7fc97406e9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc988009560 tx=0x7fc988009190 comp rx=0 tx=0).stop 2024-03-27T06:14:02.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.503+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 msgr2=0x7fc994152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.503+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc980000c00 tx=0x7fc980004730 comp rx=0 tx=0).stop 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 shutdown_connections 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc97406c530 0x7fc97406e9f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99415eb50 0x7fc994153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc99415e230 0x7fc994152d00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 --2- 172.21.15.132:0/1271515554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc9940c9570 0x7fc9941f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.507+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 >> 172.21.15.132:0/1271515554 conn(0x7fc9940c89a0 msgr2=0x7fc994070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:02.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.510+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 shutdown_connections 2024-03-27T06:14:02.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.510+0000 7fc99b400640 1 -- 172.21.15.132:0/1271515554 wait complete. 2024-03-27T06:14:02.536 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-27T06:14:02.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.726+0000 7fa2fae00640 1 Processor -- start 2024-03-27T06:14:02.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.726+0000 7fa2fae00640 1 -- start start 2024-03-27T06:14:02.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2fae00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa2f415bfa0 0x7fa2f415c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2fae00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa2f4152500 0x7fa2f4152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2fae00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f4071930 con 0x7fa2f415c380 2024-03-27T06:14:02.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2fae00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f4072110 con 0x7fa2f415bfa0 2024-03-27T06:14:02.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2fae00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f406c530 con 0x7fa2f4152500 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f3400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa2f415c380 0x7fa2f40735f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42680/0 (socket says 172.21.15.132:42680) 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f3400640 1 -- 172.21.15.132:0/2095764549 learned_addr learned my addr 172.21.15.132:0/2095764549 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f8e00640 1 --2- 172.21.15.132:0/2095764549 >> v2:172.21.15.134:3300/0 conn(0x7fa2f4152500 0x7fa2f4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f3e00640 1 --2- 172.21.15.132:0/2095764549 >> v2:172.21.15.132:3300/0 conn(0x7fa2f415bfa0 0x7fa2f415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f8e00640 1 -- 172.21.15.132:0/2095764549 >> v2:172.21.15.132:3300/0 conn(0x7fa2f415bfa0 msgr2=0x7fa2f415c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f8e00640 1 --2- 172.21.15.132:0/2095764549 >> v2:172.21.15.132:3300/0 conn(0x7fa2f415bfa0 0x7fa2f415c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f8e00640 1 -- 172.21.15.132:0/2095764549 >> v1:172.21.15.132:6789/0 conn(0x7fa2f415c380 legacy=0x7fa2f40735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f2a00640 1 -- 172.21.15.132:0/2095764549 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996687750 0 0) 0x7fa2f4071930 con 0x7fa2f415c380 2024-03-27T06:14:02.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.728+0000 7fa2f8e00640 1 -- 172.21.15.132:0/2095764549 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f4071bd0 con 0x7fa2f4152500 2024-03-27T06:14:02.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2f8e00640 1 --2- 172.21.15.132:0/2095764549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f4152500 0x7fa2f4152990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40c30d0 tx=0x7fa2ec02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc14576a069ff4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:02.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2f2a00640 1 -- 172.21.15.132:0/2095764549 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec039070 con 0x7fa2f4152500 2024-03-27T06:14:02.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2f2a00640 1 -- 172.21.15.132:0/2095764549 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec02cd00 con 0x7fa2f4152500 2024-03-27T06:14:02.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2f2a00640 1 -- 172.21.15.132:0/2095764549 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec034660 con 0x7fa2f4152500 2024-03-27T06:14:02.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 -- 172.21.15.132:0/2095764549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f4152500 msgr2=0x7fa2f4152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 --2- 172.21.15.132:0/2095764549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f4152500 0x7fa2f4152990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40c30d0 tx=0x7fa2ec02bf50 comp rx=0 tx=0).stop 2024-03-27T06:14:02.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 -- 172.21.15.132:0/2095764549 shutdown_connections 2024-03-27T06:14:02.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 --2- 172.21.15.132:0/2095764549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f4152500 0x7fa2f4152990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 --2- 172.21.15.132:0/2095764549 >> v2:172.21.15.132:3300/0 conn(0x7fa2f415bfa0 0x7fa2f415c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 -- 172.21.15.132:0/2095764549 >> 172.21.15.132:0/2095764549 conn(0x7fa2f40c89a0 msgr2=0x7fa2f40c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:02.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 -- 172.21.15.132:0/2095764549 shutdown_connections 2024-03-27T06:14:02.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.730+0000 7fa2fae00640 1 -- 172.21.15.132:0/2095764549 wait complete. 2024-03-27T06:14:02.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.732+0000 7fa2fae00640 1 Processor -- start 2024-03-27T06:14:02.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.732+0000 7fa2fae00640 1 -- start start 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2f415c380 0x7fa2f41f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4071e70 con 0x7fa2f415c380 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40c97e0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2fae00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f406ca60 con 0x7fa2f4152500 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2f3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2f3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.734+0000 7fa2f8e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2f415c380 0x7fa2f41f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:60608/0 (socket says 172.21.15.132:60608) 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 -- 172.21.15.132:0/4252908743 learned_addr learned my addr 172.21.15.132:0/4252908743 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:02.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:41026/0 (socket says 172.21.15.132:41026) 2024-03-27T06:14:02.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 -- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 msgr2=0x7fa2f415b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 -- 172.21.15.132:0/4252908743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2f415c380 msgr2=0x7fa2f41f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2f415c380 0x7fa2f41f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 -- 172.21.15.132:0/4252908743 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f40c6ad0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3e00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2f3400640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c3e0 tx=0x7fa2e000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0017070 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e0005ce0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f41f0e90 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0007310 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.735+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f41f3bd0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.737+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f41f3dd0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.738+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa2e001a070 con 0x7fa2f415bfa0 2024-03-27T06:14:02.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.738+0000 7fa2ebe00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 0x7fa2d406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:02.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.738+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fa2e008a8f0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.738+0000 7fa2f3e00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 0x7fa2d406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:02.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.738+0000 7fa2f3e00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 0x7fa2d406ea20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009f30 tx=0x7fa2e4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:02.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.744+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa2f41f3dd0 con 0x7fa2f415bfa0 2024-03-27T06:14:02.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:14:02.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:14:02.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:02.982+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa2f4068950 con 0x7fa2f415bfa0 2024-03-27T06:14:03.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.510+0000 7fa2ebe00640 1 -- 172.21.15.132:0/4252908743 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fa2e0058920 con 0x7fa2f415bfa0 2024-03-27T06:14:03.501 INFO:teuthology.orchestra.run.smithi132.stderr:new fs with metadata pool 2 and data pool 3 2024-03-27T06:14:03.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.515+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 msgr2=0x7fa2d406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.515+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 0x7fa2d406ea20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009f30 tx=0x7fa2e4009190 comp rx=0 tx=0).stop 2024-03-27T06:14:03.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.515+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 msgr2=0x7fa2f40c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.515+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c3e0 tx=0x7fa2e000c8a0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 shutdown_connections 2024-03-27T06:14:03.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2f415c380 0x7fa2f41f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa2d406c560 0x7fa2d406ea20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f415bfa0 0x7fa2f40c5230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 --2- 172.21.15.132:0/4252908743 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa2f4152500 0x7fa2f415b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.521+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 >> 172.21.15.132:0/4252908743 conn(0x7fa2f40c89a0 msgr2=0x7fa2f40713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:03.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.522+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 shutdown_connections 2024-03-27T06:14:03.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.525+0000 7fa2fae00640 1 -- 172.21.15.132:0/4252908743 wait complete. 2024-03-27T06:14:03.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:03.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.726+0000 7fd487c00640 1 Processor -- start 2024-03-27T06:14:03.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.726+0000 7fd487c00640 1 -- start start 2024-03-27T06:14:03.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd487c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd48006b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd487c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd48015b790 con 0x7fd48015aa00 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd487c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4800c12d0 con 0x7fd4800b11e0 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd487c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd4800c1080 con 0x7fd48006b520 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd47fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd48006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd485200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd4800b11e0 0x7fd4800c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55418/0 (socket says 172.21.15.132:55418) 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.727+0000 7fd485200640 1 -- 172.21.15.132:0/2673381926 learned_addr learned my addr 172.21.15.132:0/2673381926 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.728+0000 7fd47fe00640 1 -- 172.21.15.132:0/2673381926 >> v1:172.21.15.134:6789/0 conn(0x7fd4800b11e0 legacy=0x7fd4800c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.728+0000 7fd47f400640 1 -- 172.21.15.132:0/2673381926 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268485803 0 0) 0x7fd4800c12d0 con 0x7fd4800b11e0 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.728+0000 7fd47fe00640 1 -- 172.21.15.132:0/2673381926 >> v1:172.21.15.132:6789/0 conn(0x7fd48015aa00 legacy=0x7fd48015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.728+0000 7fd47fe00640 1 -- 172.21.15.132:0/2673381926 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4800c15b0 con 0x7fd48006b520 2024-03-27T06:14:03.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.729+0000 7fd47fe00640 1 --2- 172.21.15.132:0/2673381926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd48006b960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0097d0 tx=0x7fd46c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d98dfe7c8c5d7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:03.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.729+0000 7fd47f400640 1 -- 172.21.15.132:0/2673381926 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c037070 con 0x7fd48006b520 2024-03-27T06:14:03.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.729+0000 7fd47f400640 1 -- 172.21.15.132:0/2673381926 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46c030710 con 0x7fd48006b520 2024-03-27T06:14:03.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.729+0000 7fd47f400640 1 -- 172.21.15.132:0/2673381926 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c030ae0 con 0x7fd48006b520 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 -- 172.21.15.132:0/2673381926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 msgr2=0x7fd48006b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 --2- 172.21.15.132:0/2673381926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd48006b960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0097d0 tx=0x7fd46c02b800 comp rx=0 tx=0).stop 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 -- 172.21.15.132:0/2673381926 shutdown_connections 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 --2- 172.21.15.132:0/2673381926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd48006b960 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 -- 172.21.15.132:0/2673381926 >> 172.21.15.132:0/2673381926 conn(0x7fd4800cc060 msgr2=0x7fd4800cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 -- 172.21.15.132:0/2673381926 shutdown_connections 2024-03-27T06:14:03.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.730+0000 7fd487c00640 1 -- 172.21.15.132:0/2673381926 wait complete. 2024-03-27T06:14:03.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.731+0000 7fd487c00640 1 Processor -- start 2024-03-27T06:14:03.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.731+0000 7fd487c00640 1 -- start start 2024-03-27T06:14:03.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd4801511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:03.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 0x7fd4800b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:03.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd48015aa00 0x7fd4801598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:03.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800b15c0 con 0x7fd48015aa00 2024-03-27T06:14:03.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800c1c00 con 0x7fd4800b11e0 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.732+0000 7fd487c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd4800c2090 con 0x7fd48006b520 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd4801511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 0x7fd4800b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 0x7fd4800b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:60616/0 (socket says 172.21.15.132:60616) 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47fe00640 1 -- 172.21.15.132:0/1426664656 learned_addr learned my addr 172.21.15.132:0/1426664656 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd48015aa00 0x7fd4801598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 -- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 msgr2=0x7fd4800b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 0x7fd4800b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 -- 172.21.15.132:0/1426664656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd48015aa00 msgr2=0x7fd4801598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd48015aa00 0x7fd4801598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 -- 172.21.15.132:0/1426664656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4801f0b80 con 0x7fd48006b520 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd485200640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd4801511a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c780 tx=0x7fd47000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470018070 con 0x7fd48006b520 2024-03-27T06:14:03.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47000a5e0 con 0x7fd48006b520 2024-03-27T06:14:03.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd480154570 con 0x7fd48006b520 2024-03-27T06:14:03.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.733+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47000a9d0 con 0x7fd48006b520 2024-03-27T06:14:03.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.734+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd480154b30 con 0x7fd48006b520 2024-03-27T06:14:03.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.735+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd480068840 con 0x7fd48006b520 2024-03-27T06:14:03.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.735+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd47001b070 con 0x7fd48006b520 2024-03-27T06:14:03.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.736+0000 7fd47cc00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 0x7fd46006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:03.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.736+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd47008ab60 con 0x7fd48006b520 2024-03-27T06:14:03.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.736+0000 7fd47fe00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 0x7fd46006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:03.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.737+0000 7fd47fe00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 0x7fd46006ea20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009720 tx=0x7fd46c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:03.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.741+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd470058b10 con 0x7fd48006b520 2024-03-27T06:14:03.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.939+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4800c98a0 con 0x7fd48006b520 2024-03-27T06:14:03.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.940+0000 7fd47cc00640 1 -- 172.21.15.132:0/1426664656 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16658 (secure 0 0 0) 0x7fd47005ce10 con 0x7fd48006b520 2024-03-27T06:14:03.931 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:03.932 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":20,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:02.985259+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:03.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 msgr2=0x7fd46006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 0x7fd46006ea20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009720 tx=0x7fd46c008030 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 msgr2=0x7fd4801511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd4801511a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c780 tx=0x7fd47000cc40 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 shutdown_connections 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd48015aa00 0x7fd4801598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd46006c560 0x7fd46006ea20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4800b11e0 0x7fd4800b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 --2- 172.21.15.132:0/1426664656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd48006b520 0x7fd4801511a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 >> 172.21.15.132:0/1426664656 conn(0x7fd4800cc060 msgr2=0x7fd4800cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 shutdown_connections 2024-03-27T06:14:03.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:03.943+0000 7fd487c00640 1 -- 172.21.15.132:0/1426664656 wait complete. 2024-03-27T06:14:03.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:04.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.159+0000 7f1063200640 1 Processor -- start 2024-03-27T06:14:04.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.159+0000 7f1063200640 1 -- start start 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.160+0000 7f1063200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f10641549c0 0x7f106415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.160+0000 7f1063200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f106406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.160+0000 7f1063200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1064071220 con 0x7f106406bf80 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.160+0000 7f1063200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f1064071d80 con 0x7f10641549c0 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.160+0000 7f1063200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f106406d020 con 0x7f106406c7a0 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1062800640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f106406bf80 0x7f1064072210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42700/0 (socket says 172.21.15.132:42700) 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1062800640 1 -- 172.21.15.132:0/1620592063 learned_addr learned my addr 172.21.15.132:0/1620592063 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061e00640 1 --2- 172.21.15.132:0/1620592063 >> v2:172.21.15.134:3300/0 conn(0x7f10641549c0 0x7f106415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1059400640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f106406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061e00640 1 -- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 msgr2=0x7f106406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:04.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061e00640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f106406ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061e00640 1 -- 172.21.15.132:0/1620592063 >> v1:172.21.15.132:6789/0 conn(0x7f106406bf80 legacy=0x7f1064072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061e00640 1 -- 172.21.15.132:0/1620592063 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106406d300 con 0x7f10641549c0 2024-03-27T06:14:04.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.161+0000 7f1061400640 1 -- 172.21.15.132:0/1620592063 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4115228943 0 0) 0x7f105c001490 con 0x7f106406bf80 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.162+0000 7f1061e00640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10641549c0 0x7f106415b1f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10540097a0 tx=0x7f105402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6c2f1fdb6320825 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.162+0000 7f1061400640 1 -- 172.21.15.132:0/1620592063 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054034070 con 0x7f10641549c0 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.162+0000 7f1061400640 1 -- 172.21.15.132:0/1620592063 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1054030800 con 0x7f10641549c0 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.162+0000 7f1061400640 1 -- 172.21.15.132:0/1620592063 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054030bd0 con 0x7f10641549c0 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 -- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10641549c0 msgr2=0x7f106415b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10641549c0 0x7f106415b1f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10540097a0 tx=0x7f105402b940 comp rx=0 tx=0).stop 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 -- 172.21.15.132:0/1620592063 shutdown_connections 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f106406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 --2- 172.21.15.132:0/1620592063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10641549c0 0x7f106415b1f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 -- 172.21.15.132:0/1620592063 >> 172.21.15.132:0/1620592063 conn(0x7f10640cc7a0 msgr2=0x7f10640c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:04.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 -- 172.21.15.132:0/1620592063 shutdown_connections 2024-03-27T06:14:04.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.163+0000 7f1063200640 1 -- 172.21.15.132:0/1620592063 wait complete. 2024-03-27T06:14:04.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.165+0000 7f1063200640 1 Processor -- start 2024-03-27T06:14:04.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.165+0000 7f1063200640 1 -- start start 2024-03-27T06:14:04.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f1064151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 0x7f1064151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1064071a10 con 0x7f10641549c0 2024-03-27T06:14:04.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1064071730 con 0x7f106406bf80 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.166+0000 7f1063200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f106406cbc0 con 0x7f106406c7a0 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:60646/0 (socket says 172.21.15.132:60646) 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 -- 172.21.15.132:0/2520353732 learned_addr learned my addr 172.21.15.132:0/2520353732 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1059400640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f1064151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1062800640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 0x7f1064151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 -- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 msgr2=0x7f1064151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f1064151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 -- 172.21.15.132:0/2520353732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 msgr2=0x7f1064151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 0x7f1064151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 -- 172.21.15.132:0/2520353732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10641e4280 con 0x7f106406bf80 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1062800640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 0x7f1064151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1061e00640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f105402bf50 tx=0x7f10540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054048070 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1054030600 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.167+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641e5af0 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.168+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054034040 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.168+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1064151110 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.169+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10640c3e80 con 0x7f106406bf80 2024-03-27T06:14:04.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.169+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f105404b070 con 0x7f106406bf80 2024-03-27T06:14:04.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.170+0000 7f105aa00640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 0x7f104006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.170+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10540acf80 con 0x7f106406bf80 2024-03-27T06:14:04.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.171+0000 7f1059400640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 0x7f104006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.171+0000 7f1059400640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 0x7f104006e9f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1064070f90 tx=0x7f104c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.175+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f10640c3e80 con 0x7f106406bf80 2024-03-27T06:14:04.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.360+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10640c7860 con 0x7f106406bf80 2024-03-27T06:14:04.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.362+0000 7f105aa00640 1 -- 172.21.15.132:0/2520353732 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16658 (secure 0 0 0) 0x7f10640c7860 con 0x7f106406bf80 2024-03-27T06:14:04.353 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:04.354 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":20,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:02.985259+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:04.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.364+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 msgr2=0x7f104006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.364+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 0x7f104006e9f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1064070f90 tx=0x7f104c009190 comp rx=0 tx=0).stop 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.364+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 msgr2=0x7f106414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.364+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f105402bf50 tx=0x7f10540046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 shutdown_connections 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10641549c0 0x7f1064151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f106406c7a0 0x7f1064151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f104006c530 0x7f104006e9f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 --2- 172.21.15.132:0/2520353732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f106406bf80 0x7f106414c2c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 >> 172.21.15.132:0/2520353732 conn(0x7f10640cc7a0 msgr2=0x7f1064072210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:04.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 shutdown_connections 2024-03-27T06:14:04.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.365+0000 7f1063200640 1 -- 172.21.15.132:0/2520353732 wait complete. 2024-03-27T06:14:04.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-27T06:14:04.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.596+0000 7f33c7c00640 1 Processor -- start 2024-03-27T06:14:04.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.596+0000 7f33c7c00640 1 -- start start 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.597+0000 7f33c7c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f33c01524b0 0x7f33c006b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c7c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 0x7f33c006bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c7c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c00c7b10 con 0x7f33c0159430 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c7c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f33c00c8a00 con 0x7f33c01524b0 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c7c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f33c00c7df0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f33c01524b0 0x7f33c006b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33bfe00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f33c0159430 0x7f33c00c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55430/0 (socket says 172.21.15.132:55430) 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33bfe00640 1 -- 172.21.15.132:0/3178740959 learned_addr learned my addr 172.21.15.132:0/3178740959 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5c00640 1 --2- 172.21.15.132:0/3178740959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 0x7f33c006bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33bf400640 1 -- 172.21.15.132:0/3178740959 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025763680 0 0) 0x7f33c00c7b10 con 0x7f33c0159430 2024-03-27T06:14:04.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5c00640 1 -- 172.21.15.132:0/3178740959 >> v2:172.21.15.132:3300/0 conn(0x7f33c01524b0 msgr2=0x7f33c006b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:04.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5c00640 1 --2- 172.21.15.132:0/3178740959 >> v2:172.21.15.132:3300/0 conn(0x7f33c01524b0 0x7f33c006b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5c00640 1 -- 172.21.15.132:0/3178740959 >> v1:172.21.15.134:6789/0 conn(0x7f33c0159430 legacy=0x7f33c00c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.598+0000 7f33c5c00640 1 -- 172.21.15.132:0/3178740959 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c00c80d0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.599+0000 7f33c5c00640 1 --2- 172.21.15.132:0/3178740959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 0x7f33c006bd10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009d90 tx=0x7f33b802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8efb1ac8d34cf74f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33bf400640 1 -- 172.21.15.132:0/3178740959 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8039070 con 0x7f33c00ca1f0 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33bf400640 1 -- 172.21.15.132:0/3178740959 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b802cce0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33bf400640 1 -- 172.21.15.132:0/3178740959 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8034660 con 0x7f33c00ca1f0 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 -- 172.21.15.132:0/3178740959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 msgr2=0x7f33c006bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 --2- 172.21.15.132:0/3178740959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 0x7f33c006bd10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009d90 tx=0x7f33b802bf30 comp rx=0 tx=0).stop 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 -- 172.21.15.132:0/3178740959 shutdown_connections 2024-03-27T06:14:04.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 --2- 172.21.15.132:0/3178740959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c00ca1f0 0x7f33c006bd10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 --2- 172.21.15.132:0/3178740959 >> v2:172.21.15.132:3300/0 conn(0x7f33c01524b0 0x7f33c006b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 -- 172.21.15.132:0/3178740959 >> 172.21.15.132:0/3178740959 conn(0x7f33c006c630 msgr2=0x7f33c006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:04.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 -- 172.21.15.132:0/3178740959 shutdown_connections 2024-03-27T06:14:04.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.600+0000 7f33c7c00640 1 -- 172.21.15.132:0/3178740959 wait complete. 2024-03-27T06:14:04.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.602+0000 7f33c7c00640 1 Processor -- start 2024-03-27T06:14:04.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.602+0000 7f33c7c00640 1 -- start start 2024-03-27T06:14:04.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33c01524b0 0x7f33c01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c0159430 0x7f33c00c3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f33c0152890 con 0x7f33c00ca1f0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f33c00c8690 con 0x7f33c01524b0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c7c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f33c0152a10 con 0x7f33c0159430 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c5200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c5200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56656/0 (socket says 172.21.15.132:56656) 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c5200640 1 -- 172.21.15.132:0/2074709676 learned_addr learned my addr 172.21.15.132:0/2074709676 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33c5c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c0159430 0x7f33c00c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.603+0000 7f33bfe00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33c01524b0 0x7f33c01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 -- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c0159430 msgr2=0x7f33c00c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c0159430 0x7f33c00c3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 -- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33c01524b0 msgr2=0x7f33c01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33c01524b0 0x7f33c01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 -- 172.21.15.132:0/2074709676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c01f25c0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c5200640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f33b000e9e0 tx=0x7f33b000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b0017070 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b000c9d0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b000cdc0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c00c3020 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.604+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c00cb2f0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.605+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c0068950 con 0x7f33c00ca1f0 2024-03-27T06:14:04.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.606+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f33b001a070 con 0x7f33c00ca1f0 2024-03-27T06:14:04.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.607+0000 7f33bcc00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 0x7f33a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:04.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.607+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f33b008b610 con 0x7f33c00ca1f0 2024-03-27T06:14:04.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.608+0000 7f33bfe00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 0x7f33a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:04.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.608+0000 7f33bfe00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 0x7f33a006ea20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f33c01f40e0 tx=0x7f33ac007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:04.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.613+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f33b00595e0 con 0x7f33c00ca1f0 2024-03-27T06:14:04.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:04.852+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f33c00b2e50 con 0x7f33c00ca1f0 2024-03-27T06:14:05.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.527+0000 7f33bcc00640 1 -- 172.21.15.132:0/2074709676 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f33b005d8e0 con 0x7f33c00ca1f0 2024-03-27T06:14:05.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 msgr2=0x7f33a006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 0x7f33a006ea20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f33c01f40e0 tx=0x7f33ac007de0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 msgr2=0x7f33c014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f33b000e9e0 tx=0x7f33b000eea0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 shutdown_connections 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f33c0159430 0x7f33c00c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f33a006c560 0x7f33a006ea20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33c01524b0 0x7f33c01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 --2- 172.21.15.132:0/2074709676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33c00ca1f0 0x7f33c014f920 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 >> 172.21.15.132:0/2074709676 conn(0x7f33c006c630 msgr2=0x7f33c01e1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 shutdown_connections 2024-03-27T06:14:05.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.531+0000 7f33c7c00640 1 -- 172.21.15.132:0/2074709676 wait complete. 2024-03-27T06:14:05.542 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2024-03-27T06:14:05.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-27T06:14:05.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.763+0000 7fed34600640 1 Processor -- start 2024-03-27T06:14:05.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.763+0000 7fed34600640 1 -- start start 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.764+0000 7fed34600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fed2c15bfa0 0x7fed2c15c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed34600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 0x7fed2c152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed34600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed2c071930 con 0x7fed2c15c380 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed34600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fed2c072110 con 0x7fed2c15bfa0 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed34600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fed2c06c530 con 0x7fed2c152500 2024-03-27T06:14:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fed2c15bfa0 0x7fed2c15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fed2c15bfa0 0x7fed2c15c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56666/0 (socket says 172.21.15.132:56666) 2024-03-27T06:14:05.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed32600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 0x7fed2c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 -- 172.21.15.132:0/666892283 learned_addr learned my addr 172.21.15.132:0/666892283 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 -- 172.21.15.132:0/666892283 >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 msgr2=0x7fed2c152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 --2- 172.21.15.132:0/666892283 >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 0x7fed2c152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 -- 172.21.15.132:0/666892283 >> v1:172.21.15.132:6789/0 conn(0x7fed2c15c380 legacy=0x7fed2c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed31c00640 1 -- 172.21.15.132:0/666892283 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c071bd0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.765+0000 7fed32600640 1 --2- 172.21.15.132:0/666892283 >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 0x7fed2c152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:05.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed31c00640 1 --2- 172.21.15.132:0/666892283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c15c910 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0c30d0 tx=0x7fed2002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5494a95b85a4244e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:05.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed2be00640 1 -- 172.21.15.132:0/666892283 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2002f850 con 0x7fed2c15bfa0 2024-03-27T06:14:05.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed2be00640 1 -- 172.21.15.132:0/666892283 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed2002f9f0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed2be00640 1 -- 172.21.15.132:0/666892283 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2002fdc0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed34600640 1 -- 172.21.15.132:0/666892283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 msgr2=0x7fed2c15c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.767+0000 7fed34600640 1 --2- 172.21.15.132:0/666892283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c15c910 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0c30d0 tx=0x7fed2002ba30 comp rx=0 tx=0).stop 2024-03-27T06:14:05.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 -- 172.21.15.132:0/666892283 shutdown_connections 2024-03-27T06:14:05.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 --2- 172.21.15.132:0/666892283 >> v2:172.21.15.134:3300/0 conn(0x7fed2c152500 0x7fed2c152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 --2- 172.21.15.132:0/666892283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c15c910 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 -- 172.21.15.132:0/666892283 >> 172.21.15.132:0/666892283 conn(0x7fed2c0c89a0 msgr2=0x7fed2c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 -- 172.21.15.132:0/666892283 shutdown_connections 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.768+0000 7fed34600640 1 -- 172.21.15.132:0/666892283 wait complete. 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.770+0000 7fed34600640 1 Processor -- start 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.770+0000 7fed34600640 1 -- start start 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed2c152500 0x7fed2c1f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c15c380 0x7fed2c1485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c071e70 con 0x7fed2c15bfa0 2024-03-27T06:14:05.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c0c97e0 con 0x7fed2c15c380 2024-03-27T06:14:05.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed34600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fed2c06ca60 con 0x7fed2c152500 2024-03-27T06:14:05.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.771+0000 7fed31200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56668/0 (socket says 172.21.15.132:56668) 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 -- 172.21.15.132:0/2576701362 learned_addr learned my addr 172.21.15.132:0/2576701362 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed32600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c15c380 0x7fed2c1485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31c00640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed2c152500 0x7fed2c1f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 -- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed2c152500 msgr2=0x7fed2c1f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed2c152500 0x7fed2c1f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 -- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c15c380 msgr2=0x7fed2c1485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c15c380 0x7fed2c1485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 -- 172.21.15.132:0/2576701362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c0c6cf0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed31200640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c3c0 tx=0x7fed1800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18017070 con 0x7fed2c15bfa0 2024-03-27T06:14:05.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed180052f0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18003850 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.772+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c0c6730 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.773+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c1530d0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.774+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c068950 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.775+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fed1801a070 con 0x7fed2c15bfa0 2024-03-27T06:14:05.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.776+0000 7fed29600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 0x7fed0c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:05.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.776+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fed1808a8f0 con 0x7fed2c15bfa0 2024-03-27T06:14:05.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.776+0000 7fed31c00640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 0x7fed0c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:05.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.777+0000 7fed31c00640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 0x7fed0c06e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed20036000 tx=0x7fed20008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:05.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:05.780+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fed18058790 con 0x7fed2c15bfa0 2024-03-27T06:14:06.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.022+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fed2c06cd30 con 0x7fed2c15bfa0 2024-03-27T06:14:06.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.543+0000 7fed29600640 1 -- 172.21.15.132:0/2576701362 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7fed1805ca90 con 0x7fed2c15bfa0 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.546+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 msgr2=0x7fed0c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.546+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 0x7fed0c06e9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed20036000 tx=0x7fed20008030 comp rx=0 tx=0).stop 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.546+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 msgr2=0x7fed2c150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c3c0 tx=0x7fed1800c7c0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 shutdown_connections 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed0c06c530 0x7fed0c06e9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c15c380 0x7fed2c1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed2c15bfa0 0x7fed2c150de0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 --2- 172.21.15.132:0/2576701362 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed2c152500 0x7fed2c1f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 >> 172.21.15.132:0/2576701362 conn(0x7fed2c0c89a0 msgr2=0x7fed2c070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:06.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 shutdown_connections 2024-03-27T06:14:06.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.547+0000 7fed34600640 1 -- 172.21.15.132:0/2576701362 wait complete. 2024-03-27T06:14:06.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-27T06:14:06.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.787+0000 7efd30c00640 1 Processor -- start 2024-03-27T06:14:06.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.788+0000 7efd30c00640 1 -- start start 2024-03-27T06:14:06.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd30c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7efd2c0cbf40 0x7efd2c166220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:06.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd30c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2c06d440 con 0x7efd2c154db0 2024-03-27T06:14:06.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd30c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2c15b8e0 con 0x7efd2c0cd710 2024-03-27T06:14:06.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd30c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7efd2c0cdb50 con 0x7efd2c0cbf40 2024-03-27T06:14:06.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd29e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7efd2c154db0 0x7efd2c0cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42714/0 (socket says 172.21.15.132:42714) 2024-03-27T06:14:06.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.789+0000 7efd29e00640 1 -- 172.21.15.132:0/2288241577 learned_addr learned my addr 172.21.15.132:0/2288241577 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:06.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd2a800640 1 --2- 172.21.15.132:0/2288241577 >> v2:172.21.15.132:3300/0 conn(0x7efd2c0cbf40 0x7efd2c166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172565369 0 0) 0x7efd2c15b8e0 con 0x7efd2c0cd710 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd10003520 con 0x7efd2c0cd710 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4048971523 0 0) 0x7efd1c001110 con 0x7efd2c154db0 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd2c15b8e0 con 0x7efd2c154db0 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd2a800640 1 -- 172.21.15.132:0/2288241577 >> v1:172.21.15.134:6789/0 conn(0x7efd2c0cd710 legacy=0x7efd2c06c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd2a800640 1 -- 172.21.15.132:0/2288241577 >> v1:172.21.15.132:6789/0 conn(0x7efd2c154db0 legacy=0x7efd2c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.790+0000 7efd2a800640 1 -- 172.21.15.132:0/2288241577 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c0cac20 con 0x7efd2c0cbf40 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.791+0000 7efd2a800640 1 --2- 172.21.15.132:0/2288241577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c166220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd24009d90 tx=0x7efd2402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49f6e1a4b713038 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:06.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.791+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24039070 con 0x7efd2c0cbf40 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.791+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd2402cd00 con 0x7efd2c0cbf40 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd28a00640 1 -- 172.21.15.132:0/2288241577 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24034660 con 0x7efd2c0cbf40 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 -- 172.21.15.132:0/2288241577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 msgr2=0x7efd2c166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 --2- 172.21.15.132:0/2288241577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c166220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd24009d90 tx=0x7efd2402bf50 comp rx=0 tx=0).stop 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 -- 172.21.15.132:0/2288241577 shutdown_connections 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 --2- 172.21.15.132:0/2288241577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c166220 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 -- 172.21.15.132:0/2288241577 >> 172.21.15.132:0/2288241577 conn(0x7efd2c0ccbf0 msgr2=0x7efd2c0cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 -- 172.21.15.132:0/2288241577 shutdown_connections 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.792+0000 7efd30c00640 1 -- 172.21.15.132:0/2288241577 wait complete. 2024-03-27T06:14:06.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.794+0000 7efd30c00640 1 Processor -- start 2024-03-27T06:14:06.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.794+0000 7efd30c00640 1 -- start start 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efd2c0cd710 0x7efd2c14d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 0x7efd2c157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c0cc860 con 0x7efd2c0cbf40 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c0caf50 con 0x7efd2c154db0 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd30c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7efd2c15b600 con 0x7efd2c0cd710 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd29e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:06.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd29e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56686/0 (socket says 172.21.15.132:56686) 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd29e00640 1 -- 172.21.15.132:0/3302712365 learned_addr learned my addr 172.21.15.132:0/3302712365 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd2a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 0x7efd2c157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.795+0000 7efd29400640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efd2c0cd710 0x7efd2c14d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 -- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efd2c0cd710 msgr2=0x7efd2c14d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efd2c0cd710 0x7efd2c14d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 -- 172.21.15.132:0/3302712365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 msgr2=0x7efd2c1f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c1f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 -- 172.21.15.132:0/3302712365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c1589a0 con 0x7efd2c154db0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd2a800640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 0x7efd2c157770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd24033000 tx=0x7efd24004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24039050 con 0x7efd2c154db0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd24045070 con 0x7efd2c154db0 2024-03-27T06:14:06.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.796+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24009450 con 0x7efd2c154db0 2024-03-27T06:14:06.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.797+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c158c80 con 0x7efd2c154db0 2024-03-27T06:14:06.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.797+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c150350 con 0x7efd2c154db0 2024-03-27T06:14:06.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.798+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2c068950 con 0x7efd2c154db0 2024-03-27T06:14:06.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.799+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7efd2403e870 con 0x7efd2c154db0 2024-03-27T06:14:06.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.800+0000 7efd22000640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 0x7efd0806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:06.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.800+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efd240adf90 con 0x7efd2c154db0 2024-03-27T06:14:06.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.800+0000 7efd29e00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 0x7efd0806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:06.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.801+0000 7efd29e00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 0x7efd0806e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0012b0 tx=0x7efd1c007bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:06.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:06.804+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7efd2404f020 con 0x7efd2c154db0 2024-03-27T06:14:07.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.052+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7efd2c0cbd20 con 0x7efd2c154db0 2024-03-27T06:14:07.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.555+0000 7efd22000640 1 -- 172.21.15.132:0/3302712365 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7efd2407bec0 con 0x7efd2c154db0 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.558+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 msgr2=0x7efd0806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.558+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 0x7efd0806e9f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0012b0 tx=0x7efd1c007bd0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 msgr2=0x7efd2c157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 0x7efd2c157770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd24033000 tx=0x7efd24004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 shutdown_connections 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efd0806c530 0x7efd0806e9f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c154db0 0x7efd2c157770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efd2c0cd710 0x7efd2c14d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 --2- 172.21.15.132:0/3302712365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd2c0cbf40 0x7efd2c1f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 >> 172.21.15.132:0/3302712365 conn(0x7efd2c0ccbf0 msgr2=0x7efd2c153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:07.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 shutdown_connections 2024-03-27T06:14:07.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.559+0000 7efd30c00640 1 -- 172.21.15.132:0/3302712365 wait complete. 2024-03-27T06:14:07.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-27T06:14:07.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.793+0000 7fd00ac00640 1 Processor -- start 2024-03-27T06:14:07.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.793+0000 7fd00ac00640 1 -- start start 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd00ac00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd0040cc2b0 0x7fd0040d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd00ac00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0040cd8d0 con 0x7fd0040c5110 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd00ac00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0040cafe0 con 0x7fd0041517c0 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd00ac00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd0040cc770 con 0x7fd0040cc2b0 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd008e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd0040cc2b0 0x7fd0040d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd009800640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd0041517c0 0x7fd0040cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55448/0 (socket says 172.21.15.132:55448) 2024-03-27T06:14:07.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.795+0000 7fd009800640 1 -- 172.21.15.132:0/3747986302 learned_addr learned my addr 172.21.15.132:0/3747986302 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61825931 0 0) 0x7fd0040cafe0 con 0x7fd0041517c0 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfec003520 con 0x7fd0041517c0 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd008e00640 1 -- 172.21.15.132:0/3747986302 >> v1:172.21.15.134:6789/0 conn(0x7fd0041517c0 legacy=0x7fd0040cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd008e00640 1 -- 172.21.15.132:0/3747986302 >> v1:172.21.15.132:6789/0 conn(0x7fd0040c5110 legacy=0x7fd004165090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619318436 0 0) 0x7fd0040cd8d0 con 0x7fd0040c5110 2024-03-27T06:14:07.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.796+0000 7fd008e00640 1 -- 172.21.15.132:0/3747986302 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0040cb2c0 con 0x7fd0040cc2b0 2024-03-27T06:14:07.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.797+0000 7fd008e00640 1 --2- 172.21.15.132:0/3747986302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0040cc2b0 0x7fd0040d0500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcff40097d0 tx=0x7fcff402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb79fc8b9d9d240 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:07.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.797+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4034070 con 0x7fd0040cc2b0 2024-03-27T06:14:07.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.797+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff4030610 con 0x7fd0040cc2b0 2024-03-27T06:14:07.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.797+0000 7fd003e00640 1 -- 172.21.15.132:0/3747986302 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40309c0 con 0x7fd0040cc2b0 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 -- 172.21.15.132:0/3747986302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0040cc2b0 msgr2=0x7fd0040d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 --2- 172.21.15.132:0/3747986302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0040cc2b0 0x7fd0040d0500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcff40097d0 tx=0x7fcff402b960 comp rx=0 tx=0).stop 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 -- 172.21.15.132:0/3747986302 shutdown_connections 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 --2- 172.21.15.132:0/3747986302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0040cc2b0 0x7fd0040d0500 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 -- 172.21.15.132:0/3747986302 >> 172.21.15.132:0/3747986302 conn(0x7fd00406dc80 msgr2=0x7fd00406e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 -- 172.21.15.132:0/3747986302 shutdown_connections 2024-03-27T06:14:07.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.798+0000 7fd00ac00640 1 -- 172.21.15.132:0/3747986302 wait complete. 2024-03-27T06:14:07.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.799+0000 7fd00ac00640 1 Processor -- start 2024-03-27T06:14:07.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.800+0000 7fd00ac00640 1 -- start start 2024-03-27T06:14:07.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.800+0000 7fd00ac00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0040c5110 0x7fd00414cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:07.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.800+0000 7fd00ac00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd0040cc2b0 0x7fd00414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00ac00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00ac00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0040c5660 con 0x7fd0041517c0 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00ac00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0040ccd30 con 0x7fd0040c5110 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00ac00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd00415e710 con 0x7fd0040cc2b0 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56714/0 (socket says 172.21.15.132:56714) 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd009800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0040c5110 0x7fd00414cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 -- 172.21.15.132:0/2706771142 learned_addr learned my addr 172.21.15.132:0/2706771142 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd008e00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd0040cc2b0 0x7fd00414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 -- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd0040cc2b0 msgr2=0x7fd00414a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd0040cc2b0 0x7fd00414a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 -- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0040c5110 msgr2=0x7fd00414cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0040c5110 0x7fd00414cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:07.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 -- 172.21.15.132:0/2706771142 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0041f0230 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.801+0000 7fd00a200640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c760 tx=0x7fcffc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.802+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc003da0 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.802+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcffc00ce10 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.802+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0041f0510 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.802+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0052b0 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.802+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0041f0ad0 con 0x7fd0041517c0 2024-03-27T06:14:07.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.803+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0040686c0 con 0x7fd0041517c0 2024-03-27T06:14:07.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.804+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcffc005450 con 0x7fd0041517c0 2024-03-27T06:14:07.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.805+0000 7fd001600640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 0x7fcfe806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:07.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.805+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcffc08b460 con 0x7fd0041517c0 2024-03-27T06:14:07.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.806+0000 7fd009800640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 0x7fcfe806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:07.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.806+0000 7fd009800640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 0x7fcfe806ea40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005880 tx=0x7fcff8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:07.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:07.810+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fcffc059400 con 0x7fd0041517c0 2024-03-27T06:14:08.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.053+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd00415b680 con 0x7fd0041517c0 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.556+0000 7fd001600640 1 -- 172.21.15.132:0/2706771142 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7fcffc05d700 con 0x7fd0041517c0 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 msgr2=0x7fcfe806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 0x7fcfe806ea40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005880 tx=0x7fcff8009040 comp rx=0 tx=0).stop 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 msgr2=0x7fd00414aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c760 tx=0x7fcffc00cc20 comp rx=0 tx=0).stop 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 shutdown_connections 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041517c0 0x7fd00414aed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd0040cc2b0 0x7fd00414a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcfe806c580 0x7fcfe806ea40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 --2- 172.21.15.132:0/2706771142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0040c5110 0x7fd00414cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.559+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 >> 172.21.15.132:0/2706771142 conn(0x7fd00406dc80 msgr2=0x7fd00415d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:08.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.560+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 shutdown_connections 2024-03-27T06:14:08.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.560+0000 7fd00ac00640 1 -- 172.21.15.132:0/2706771142 wait complete. 2024-03-27T06:14:08.573 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-27T06:14:08.573 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:08.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.780+0000 7f0fc3e00640 1 Processor -- start 2024-03-27T06:14:08.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.780+0000 7f0fc3e00640 1 -- start start 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc3e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0fc40c5640 0x7f0fc40b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc3e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc40c80f0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc3e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc40c5cb0 con 0x7f0fc4152740 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc3e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc406cc40 con 0x7f0fc40c5640 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc2000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0fc40c5640 0x7f0fc40b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc2000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0fc40c5640 0x7f0fc40b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56722/0 (socket says 172.21.15.132:56722) 2024-03-27T06:14:08.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.782+0000 7f0fc2000640 1 -- 172.21.15.132:0/3610902036 learned_addr learned my addr 172.21.15.132:0/3610902036 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:08.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.783+0000 7f0fc2000640 1 -- 172.21.15.132:0/3610902036 >> v1:172.21.15.134:6789/0 conn(0x7f0fc4152740 legacy=0x7f0fc406c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.783+0000 7f0fc2000640 1 -- 172.21.15.132:0/3610902036 >> v1:172.21.15.132:6789/0 conn(0x7f0fc4072ff0 legacy=0x7f0fc4162e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.783+0000 7f0fc2000640 1 -- 172.21.15.132:0/3610902036 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc40c5f90 con 0x7f0fc40c5640 2024-03-27T06:14:08.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc2000640 1 --2- 172.21.15.132:0/3610902036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc40c5640 0x7f0fc40b1970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009870 tx=0x7f0fac02bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff4401b996c003f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:08.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc1600640 1 -- 172.21.15.132:0/3610902036 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac02d870 con 0x7f0fc40c5640 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc1600640 1 -- 172.21.15.132:0/3610902036 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac02da10 con 0x7f0fc40c5640 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc1600640 1 -- 172.21.15.132:0/3610902036 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac02dde0 con 0x7f0fc40c5640 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc3e00640 1 -- 172.21.15.132:0/3610902036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc40c5640 msgr2=0x7f0fc40b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.784+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/3610902036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc40c5640 0x7f0fc40b1970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009870 tx=0x7f0fac02bab0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.785+0000 7f0fc3e00640 1 -- 172.21.15.132:0/3610902036 shutdown_connections 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.785+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/3610902036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc40c5640 0x7f0fc40b1970 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.785+0000 7f0fc3e00640 1 -- 172.21.15.132:0/3610902036 >> 172.21.15.132:0/3610902036 conn(0x7f0fc40cc7a0 msgr2=0x7f0fc40c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.785+0000 7f0fc3e00640 1 -- 172.21.15.132:0/3610902036 shutdown_connections 2024-03-27T06:14:08.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.785+0000 7f0fc3e00640 1 -- 172.21.15.132:0/3610902036 wait complete. 2024-03-27T06:14:08.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.786+0000 7f0fc3e00640 1 Processor -- start 2024-03-27T06:14:08.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.786+0000 7f0fc3e00640 1 -- start start 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fc40c5640 0x7f0fc41e2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fc4152740 0x7f0fc41f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4158db0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4158ad0 con 0x7f0fc4152740 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.787+0000 7f0fc3e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc4152b20 con 0x7f0fc40c5640 2024-03-27T06:14:08.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56736/0 (socket says 172.21.15.132:56736) 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 -- 172.21.15.132:0/2338811109 learned_addr learned my addr 172.21.15.132:0/2338811109 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2000640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fc40c5640 0x7f0fc41e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc3400640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fc4152740 0x7f0fc41f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 -- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fc40c5640 msgr2=0x7f0fc41e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fc40c5640 0x7f0fc41e2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 -- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fc4152740 msgr2=0x7f0fc41f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fc4152740 0x7f0fc41f0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 -- 172.21.15.132:0/2338811109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc41e6ad0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc2a00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400c3c0 tx=0x7f0fb400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4017070 con 0x7f0fc4072ff0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb40052d0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.788+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc4151c60 con 0x7f0fc4072ff0 2024-03-27T06:14:08.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.789+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4003830 con 0x7f0fc4072ff0 2024-03-27T06:14:08.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.789+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc41e1ed0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.790+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc40c31f0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.791+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0fb401a070 con 0x7f0fc4072ff0 2024-03-27T06:14:08.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.791+0000 7f0fbb400640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 0x7f0fa006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:08.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.792+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0fb408a9a0 con 0x7f0fc4072ff0 2024-03-27T06:14:08.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.792+0000 7f0fc2000640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 0x7f0fa006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:08.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.792+0000 7f0fc2000640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 0x7f0fa006e9f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036000 tx=0x7f0fac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:08.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:08.797+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f0fb40588d0 con 0x7f0fc4072ff0 2024-03-27T06:14:09.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.038+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fc414f850 con 0x7f0fc4072ff0 2024-03-27T06:14:09.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.039+0000 7f0fbb400640 1 -- 172.21.15.132:0/2338811109 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6855 (secure 0 0 0) 0x7f0fb405cbd0 con 0x7f0fc4072ff0 2024-03-27T06:14:09.030 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:09.030 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:08.560880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4273,"mds_1":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1}]} 2024-03-27T06:14:09.030 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 9 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 msgr2=0x7f0fa006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 0x7f0fa006e9f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036000 tx=0x7f0fac008030 comp rx=0 tx=0).stop 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 msgr2=0x7f0fc41e41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400c3c0 tx=0x7f0fb400c880 comp rx=0 tx=0).stop 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 shutdown_connections 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fa006c530 0x7f0fa006e9f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fc4152740 0x7f0fc41f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fc40c5640 0x7f0fc41e2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 --2- 172.21.15.132:0/2338811109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fc4072ff0 0x7f0fc41e41a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 >> 172.21.15.132:0/2338811109 conn(0x7f0fc40cc7a0 msgr2=0x7f0fc414eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:09.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 shutdown_connections 2024-03-27T06:14:09.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.042+0000 7f0fc3e00640 1 -- 172.21.15.132:0/2338811109 wait complete. 2024-03-27T06:14:09.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:09.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.267+0000 7fc567200640 1 Processor -- start 2024-03-27T06:14:09.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.267+0000 7fc567200640 1 -- start start 2024-03-27T06:14:09.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.268+0000 7fc567200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc560158f10 0x7fc56015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.268+0000 7fc567200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 0x7fc56006d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.268+0000 7fc567200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5600c1120 con 0x7fc56006b520 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.268+0000 7fc567200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fc56015fc20 con 0x7fc560158f10 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.268+0000 7fc567200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc5600c13e0 con 0x7fc56006d3e0 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc560158f10 0x7fc56015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc560158f10 0x7fc56015f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56742/0 (socket says 172.21.15.132:56742) 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 -- 172.21.15.132:0/449600446 learned_addr learned my addr 172.21.15.132:0/449600446 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:09.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc557400640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 0x7fc56006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 -- 172.21.15.132:0/449600446 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 msgr2=0x7fc56006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 0x7fc56006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55f400640 1 -- 172.21.15.132:0/449600446 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183530763 0 0) 0x7fc5600c1120 con 0x7fc56006b520 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 -- 172.21.15.132:0/449600446 >> v1:172.21.15.134:6789/0 conn(0x7fc56006b520 legacy=0x7fc56006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc55fe00640 1 -- 172.21.15.132:0/449600446 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5600c16c0 con 0x7fc560158f10 2024-03-27T06:14:09.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.269+0000 7fc557400640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 0x7fc56006d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.271+0000 7fc55fe00640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc560158f10 0x7fc56015f6e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc550009870 tx=0x7fc55002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b799501a8895d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.271+0000 7fc55f400640 1 -- 172.21.15.132:0/449600446 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550034070 con 0x7fc560158f10 2024-03-27T06:14:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.271+0000 7fc55f400640 1 -- 172.21.15.132:0/449600446 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc550030840 con 0x7fc560158f10 2024-03-27T06:14:09.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.271+0000 7fc55f400640 1 -- 172.21.15.132:0/449600446 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550030c10 con 0x7fc560158f10 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 -- 172.21.15.132:0/449600446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc560158f10 msgr2=0x7fc56015f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc560158f10 0x7fc56015f6e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc550009870 tx=0x7fc55002b980 comp rx=0 tx=0).stop 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 -- 172.21.15.132:0/449600446 shutdown_connections 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006d3e0 0x7fc56006d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 --2- 172.21.15.132:0/449600446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc560158f10 0x7fc56015f6e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 -- 172.21.15.132:0/449600446 >> 172.21.15.132:0/449600446 conn(0x7fc5600cc060 msgr2=0x7fc5600cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 -- 172.21.15.132:0/449600446 shutdown_connections 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.272+0000 7fc567200640 1 -- 172.21.15.132:0/449600446 wait complete. 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.274+0000 7fc567200640 1 Processor -- start 2024-03-27T06:14:09.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.274+0000 7fc567200640 1 -- start start 2024-03-27T06:14:09.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006b520 0x7fc560159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 0x7fc560159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600c1f60 con 0x7fc56006d3e0 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600c1c80 con 0x7fc560158f10 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.275+0000 7fc567200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc56006cec0 con 0x7fc56006b520 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56756/0 (socket says 172.21.15.132:56756) 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 -- 172.21.15.132:0/2272924049 learned_addr learned my addr 172.21.15.132:0/2272924049 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc55fe00640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006b520 0x7fc560159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc565200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 0x7fc560159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 -- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006b520 msgr2=0x7fc560159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006b520 0x7fc560159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 -- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 msgr2=0x7fc560159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 0x7fc560159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc557400640 1 -- 172.21.15.132:0/2272924049 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5601f8920 con 0x7fc56006d3e0 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.276+0000 7fc565200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 0x7fc560159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:09.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc557400640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc54800ec10 tx=0x7fc54800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548017070 con 0x7fc56006d3e0 2024-03-27T06:14:09.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc54800cc30 con 0x7fc56006d3e0 2024-03-27T06:14:09.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548013470 con 0x7fc56006d3e0 2024-03-27T06:14:09.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5601f4980 con 0x7fc56006d3e0 2024-03-27T06:14:09.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.277+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5601f4f40 con 0x7fc56006d3e0 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.278+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc560068840 con 0x7fc56006d3e0 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.279+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc54801a070 con 0x7fc56006d3e0 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.280+0000 7fc55cc00640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 0x7fc53c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.280+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc54808b9a0 con 0x7fc56006d3e0 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.280+0000 7fc55fe00640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 0x7fc53c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.281+0000 7fc55fe00640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 0x7fc53c06e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc550036000 tx=0x7fc55003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.285+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc5480598d0 con 0x7fc56006d3e0 2024-03-27T06:14:09.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.486+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc56006c570 con 0x7fc56006d3e0 2024-03-27T06:14:09.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.489+0000 7fc55cc00640 1 -- 172.21.15.132:0/2272924049 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fc54805dbd0 con 0x7fc56006d3e0 2024-03-27T06:14:09.481 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:09.481 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:09.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.492+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 msgr2=0x7fc53c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.492+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 0x7fc53c06e9f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc550036000 tx=0x7fc55003b040 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.492+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 msgr2=0x7fc5600c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.492+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc54800ec10 tx=0x7fc54800c520 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 shutdown_connections 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc53c06c530 0x7fc53c06e9f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc560158f10 0x7fc560159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc56006d3e0 0x7fc5600c9ff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 --2- 172.21.15.132:0/2272924049 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc56006b520 0x7fc560159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 >> 172.21.15.132:0/2272924049 conn(0x7fc5600cc060 msgr2=0x7fc5600ca5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 shutdown_connections 2024-03-27T06:14:09.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.493+0000 7fc567200640 1 -- 172.21.15.132:0/2272924049 wait complete. 2024-03-27T06:14:09.501 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-27T06:14:09.501 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:09.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.718+0000 7ff49bc00640 1 Processor -- start 2024-03-27T06:14:09.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.718+0000 7ff49bc00640 1 -- start start 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.719+0000 7ff49bc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff4940c8160 0x7ff494150df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.719+0000 7ff49bc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff49406b520 0x7ff4940c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff49bc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff494157a60 con 0x7ff49406b940 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff49bc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff49406e6e0 con 0x7ff49406b520 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff49bc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7ff49406cec0 con 0x7ff4940c8160 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff49406b520 0x7ff4940c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff49406b520 0x7ff4940c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56762/0 (socket says 172.21.15.132:56762) 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 -- 172.21.15.132:0/3803593012 learned_addr learned my addr 172.21.15.132:0/3803593012 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff499200640 1 --2- 172.21.15.132:0/3803593012 >> v2:172.21.15.134:3300/0 conn(0x7ff4940c8160 0x7ff494150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 -- 172.21.15.132:0/3803593012 >> v2:172.21.15.134:3300/0 conn(0x7ff4940c8160 msgr2=0x7ff494150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 --2- 172.21.15.132:0/3803593012 >> v2:172.21.15.134:3300/0 conn(0x7ff4940c8160 0x7ff494150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 -- 172.21.15.132:0/3803593012 >> v1:172.21.15.134:6789/0 conn(0x7ff49406b940 legacy=0x7ff4940c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493e00640 1 -- 172.21.15.132:0/3803593012 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49406d110 con 0x7ff49406b520 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.720+0000 7ff493400640 1 -- 172.21.15.132:0/3803593012 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132645299 0 0) 0x7ff494157a60 con 0x7ff49406b940 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.721+0000 7ff493e00640 1 --2- 172.21.15.132:0/3803593012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff4940c1080 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff49406c7b0 tx=0x7ff48002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1326432d9d7008f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff493400640 1 -- 172.21.15.132:0/3803593012 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48003b070 con 0x7ff49406b520 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff493400640 1 -- 172.21.15.132:0/3803593012 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff480036440 con 0x7ff49406b520 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff493400640 1 -- 172.21.15.132:0/3803593012 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480036810 con 0x7ff49406b520 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 -- 172.21.15.132:0/3803593012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 msgr2=0x7ff4940c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 --2- 172.21.15.132:0/3803593012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff4940c1080 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff49406c7b0 tx=0x7ff48002e170 comp rx=0 tx=0).stop 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 -- 172.21.15.132:0/3803593012 shutdown_connections 2024-03-27T06:14:09.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 --2- 172.21.15.132:0/3803593012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff4940c1080 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 --2- 172.21.15.132:0/3803593012 >> v2:172.21.15.134:3300/0 conn(0x7ff4940c8160 0x7ff494150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 -- 172.21.15.132:0/3803593012 >> 172.21.15.132:0/3803593012 conn(0x7ff4940cc060 msgr2=0x7ff4940cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 -- 172.21.15.132:0/3803593012 shutdown_connections 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.722+0000 7ff49bc00640 1 -- 172.21.15.132:0/3803593012 wait complete. 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.724+0000 7ff49bc00640 1 Processor -- start 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.724+0000 7ff49bc00640 1 -- start start 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff49406b940 0x7ff4940cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4940c8160 0x7ff4941ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff49406d6d0 con 0x7ff49406b520 2024-03-27T06:14:09.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff49406da40 con 0x7ff49406b940 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff49bc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff494151330 con 0x7ff4940c8160 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff499200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff499200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56768/0 (socket says 172.21.15.132:56768) 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff499200640 1 -- 172.21.15.132:0/518298897 learned_addr learned my addr 172.21.15.132:0/518298897 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.725+0000 7ff493e00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff49406b940 0x7ff4940cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499c00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4940c8160 0x7ff4941ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 -- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4940c8160 msgr2=0x7ff4941ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4940c8160 0x7ff4941ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 -- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff49406b940 msgr2=0x7ff4940cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff49406b940 0x7ff4940cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 -- 172.21.15.132:0/518298897 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4940c3490 con 0x7ff49406b520 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff499200640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e870 tx=0x7ff48800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488017070 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48800c940 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48800cd30 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.726+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4940c6450 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.727+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4940c3c20 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.727+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4940c2090 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.728+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff48801a070 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.729+0000 7ff490c00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 0x7ff47406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.729+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff48808b580 con 0x7ff49406b520 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.730+0000 7ff493e00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 0x7ff47406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.731+0000 7ff493e00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 0x7ff47406e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff480038000 tx=0x7ff480032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:09.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.734+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff4940c2090 con 0x7ff49406b520 2024-03-27T06:14:09.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.928+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff494068840 con 0x7ff47406c530 2024-03-27T06:14:09.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.936+0000 7ff490c00640 1 -- 172.21.15.132:0/518298897 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251871 (secure 0 0 0) 0x7ff494068840 con 0x7ff47406c530 2024-03-27T06:14:09.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:09.932 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 msgr2=0x7ff47406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 0x7ff47406e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff480038000 tx=0x7ff480032000 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 msgr2=0x7ff494140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e870 tx=0x7ff48800ed30 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 shutdown_connections 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4940c8160 0x7ff4941ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff47406c530 0x7ff47406e9f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff49406b940 0x7ff4940cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 --2- 172.21.15.132:0/518298897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff49406b520 0x7ff494140d10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 >> 172.21.15.132:0/518298897 conn(0x7ff4940cc060 msgr2=0x7ff49414f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:09.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 shutdown_connections 2024-03-27T06:14:09.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:09.942+0000 7ff49bc00640 1 -- 172.21.15.132:0/518298897 wait complete. 2024-03-27T06:14:09.952 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-27T06:14:08.392176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217960,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756760,"statfs":{"total":773094113280,"available":772870922240,"internally_reserved":0,"allocated":1794048,"data_stored":1747776,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001365"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561103+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:03.561103+0000","last_peered":"2024-03-27T06:14:03.561103+0000","last_clean":"2024-03-27T06:14:03.561103+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:03.561103+0000","last_undegraded":"2024-03-27T06:14:03.561103+0000","last_fullsized":"2024-03-27T06:14:03.561103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535967+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:03.535967+0000","last_peered":"2024-03-27T06:14:03.535967+0000","last_clean":"2024-03-27T06:14:03.535967+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:03.535967+0000","last_undegraded":"2024-03-27T06:14:03.535967+0000","last_fullsized":"2024-03-27T06:14:03.535967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536186+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:03.536186+0000","last_peered":"2024-03-27T06:14:03.536186+0000","last_clean":"2024-03-27T06:14:03.536186+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:03.536186+0000","last_undegraded":"2024-03-27T06:14:03.536186+0000","last_fullsized":"2024-03-27T06:14:03.536186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.538428+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:03.538428+0000","last_peered":"2024-03-27T06:14:03.538428+0000","last_clean":"2024-03-27T06:14:03.538428+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:03.538428+0000","last_undegraded":"2024-03-27T06:14:03.538428+0000","last_fullsized":"2024-03-27T06:14:03.538428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:53.372177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556789+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:03.556789+0000","last_peered":"2024-03-27T06:14:03.556789+0000","last_clean":"2024-03-27T06:14:03.556789+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:03.556789+0000","last_undegraded":"2024-03-27T06:14:03.556789+0000","last_fullsized":"2024-03-27T06:14:03.556789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.543307+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:03.543307+0000","last_peered":"2024-03-27T06:14:03.543307+0000","last_clean":"2024-03-27T06:14:03.543307+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:03.543307+0000","last_undegraded":"2024-03-27T06:14:03.543307+0000","last_fullsized":"2024-03-27T06:14:03.543307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:42.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552053+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:03.552053+0000","last_peered":"2024-03-27T06:14:03.552053+0000","last_clean":"2024-03-27T06:14:03.552053+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:03.552053+0000","last_undegraded":"2024-03-27T06:14:03.552053+0000","last_fullsized":"2024-03-27T06:14:03.552053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546650+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:03.546650+0000","last_peered":"2024-03-27T06:14:03.546650+0000","last_clean":"2024-03-27T06:14:03.546650+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:03.546650+0000","last_undegraded":"2024-03-27T06:14:03.546650+0000","last_fullsized":"2024-03-27T06:14:03.546650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.545757+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:03.545757+0000","last_peered":"2024-03-27T06:14:03.545757+0000","last_clean":"2024-03-27T06:14:03.545757+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:03.545757+0000","last_undegraded":"2024-03-27T06:14:03.545757+0000","last_fullsized":"2024-03-27T06:14:03.545757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.558252+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:03.558252+0000","last_peered":"2024-03-27T06:14:03.558252+0000","last_clean":"2024-03-27T06:14:03.558252+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:03.558252+0000","last_undegraded":"2024-03-27T06:14:03.558252+0000","last_fullsized":"2024-03-27T06:14:03.558252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557622+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:03.557622+0000","last_peered":"2024-03-27T06:14:03.557622+0000","last_clean":"2024-03-27T06:14:03.557622+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:03.557622+0000","last_undegraded":"2024-03-27T06:14:03.557622+0000","last_fullsized":"2024-03-27T06:14:03.557622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.559083+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:03.559083+0000","last_peered":"2024-03-27T06:14:03.559083+0000","last_clean":"2024-03-27T06:14:03.559083+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:03.559083+0000","last_undegraded":"2024-03-27T06:14:03.559083+0000","last_fullsized":"2024-03-27T06:14:03.559083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.554961+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:03.554961+0000","last_peered":"2024-03-27T06:14:03.554961+0000","last_clean":"2024-03-27T06:14:03.554961+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:03.554961+0000","last_undegraded":"2024-03-27T06:14:03.554961+0000","last_fullsized":"2024-03-27T06:14:03.554961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523526+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:03.523526+0000","last_peered":"2024-03-27T06:14:03.523526+0000","last_clean":"2024-03-27T06:14:03.523526+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:03.523526+0000","last_undegraded":"2024-03-27T06:14:03.523526+0000","last_fullsized":"2024-03-27T06:14:03.523526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552785+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:03.552785+0000","last_peered":"2024-03-27T06:14:03.552785+0000","last_clean":"2024-03-27T06:14:03.552785+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:03.552785+0000","last_undegraded":"2024-03-27T06:14:03.552785+0000","last_fullsized":"2024-03-27T06:14:03.552785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556728+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:03.556728+0000","last_peered":"2024-03-27T06:14:03.556728+0000","last_clean":"2024-03-27T06:14:03.556728+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:03.556728+0000","last_undegraded":"2024-03-27T06:14:03.556728+0000","last_fullsized":"2024-03-27T06:14:03.556728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535810+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:03.535810+0000","last_peered":"2024-03-27T06:14:03.535810+0000","last_clean":"2024-03-27T06:14:03.535810+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:03.535810+0000","last_undegraded":"2024-03-27T06:14:03.535810+0000","last_fullsized":"2024-03-27T06:14:03.535810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561781+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:03.561781+0000","last_peered":"2024-03-27T06:14:03.561781+0000","last_clean":"2024-03-27T06:14:03.561781+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:03.561781+0000","last_undegraded":"2024-03-27T06:14:03.561781+0000","last_fullsized":"2024-03-27T06:14:03.561781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536210+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:03.536210+0000","last_peered":"2024-03-27T06:14:03.536210+0000","last_clean":"2024-03-27T06:14:03.536210+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:03.536210+0000","last_undegraded":"2024-03-27T06:14:03.536210+0000","last_fullsized":"2024-03-27T06:14:03.536210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541584+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:03.541584+0000","last_peered":"2024-03-27T06:14:03.541584+0000","last_clean":"2024-03-27T06:14:03.541584+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:03.541584+0000","last_undegraded":"2024-03-27T06:14:03.541584+0000","last_fullsized":"2024-03-27T06:14:03.541584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548022+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:03.548022+0000","last_peered":"2024-03-27T06:14:03.548022+0000","last_clean":"2024-03-27T06:14:03.548022+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:03.548022+0000","last_undegraded":"2024-03-27T06:14:03.548022+0000","last_fullsized":"2024-03-27T06:14:03.548022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523683+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:03.523683+0000","last_peered":"2024-03-27T06:14:03.523683+0000","last_clean":"2024-03-27T06:14:03.523683+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:03.523683+0000","last_undegraded":"2024-03-27T06:14:03.523683+0000","last_fullsized":"2024-03-27T06:14:03.523683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.562803+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:03.562803+0000","last_peered":"2024-03-27T06:14:03.562803+0000","last_clean":"2024-03-27T06:14:03.562803+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:03.562803+0000","last_undegraded":"2024-03-27T06:14:03.562803+0000","last_fullsized":"2024-03-27T06:14:03.562803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536030+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:03.536030+0000","last_peered":"2024-03-27T06:14:03.536030+0000","last_clean":"2024-03-27T06:14:03.536030+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:03.536030+0000","last_undegraded":"2024-03-27T06:14:03.536030+0000","last_fullsized":"2024-03-27T06:14:03.536030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.530720+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:03.530720+0000","last_peered":"2024-03-27T06:14:03.530720+0000","last_clean":"2024-03-27T06:14:03.530720+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:03.530720+0000","last_undegraded":"2024-03-27T06:14:03.530720+0000","last_fullsized":"2024-03-27T06:14:03.530720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:08:35.447018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536752+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:03.536752+0000","last_peered":"2024-03-27T06:14:03.536752+0000","last_clean":"2024-03-27T06:14:03.536752+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:03.536752+0000","last_undegraded":"2024-03-27T06:14:03.536752+0000","last_fullsized":"2024-03-27T06:14:03.536752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:53.372177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516878+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:03.516878+0000","last_peered":"2024-03-27T06:14:03.516878+0000","last_clean":"2024-03-27T06:14:03.516878+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:03.516878+0000","last_undegraded":"2024-03-27T06:14:03.516878+0000","last_fullsized":"2024-03-27T06:14:03.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552275+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:03.552275+0000","last_peered":"2024-03-27T06:14:03.552275+0000","last_clean":"2024-03-27T06:14:03.552275+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:03.552275+0000","last_undegraded":"2024-03-27T06:14:03.552275+0000","last_fullsized":"2024-03-27T06:14:03.552275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530565+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:04.530565+0000","last_peered":"2024-03-27T06:14:04.530565+0000","last_clean":"2024-03-27T06:14:04.530565+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:04.530565+0000","last_undegraded":"2024-03-27T06:14:04.530565+0000","last_fullsized":"2024-03-27T06:14:04.530565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.512599+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:03.512599+0000","last_peered":"2024-03-27T06:14:03.512599+0000","last_clean":"2024-03-27T06:14:03.512599+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:03.512599+0000","last_undegraded":"2024-03-27T06:14:03.512599+0000","last_fullsized":"2024-03-27T06:14:03.512599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536047+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:03.536047+0000","last_peered":"2024-03-27T06:14:03.536047+0000","last_clean":"2024-03-27T06:14:03.536047+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:03.536047+0000","last_undegraded":"2024-03-27T06:14:03.536047+0000","last_fullsized":"2024-03-27T06:14:03.536047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529878+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:04.529878+0000","last_peered":"2024-03-27T06:14:04.529878+0000","last_clean":"2024-03-27T06:14:04.529878+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:04.529878+0000","last_undegraded":"2024-03-27T06:14:04.529878+0000","last_fullsized":"2024-03-27T06:14:04.529878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536255+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:03.536255+0000","last_peered":"2024-03-27T06:14:03.536255+0000","last_clean":"2024-03-27T06:14:03.536255+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:03.536255+0000","last_undegraded":"2024-03-27T06:14:03.536255+0000","last_fullsized":"2024-03-27T06:14:03.536255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.524898+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:03.524898+0000","last_peered":"2024-03-27T06:14:03.524898+0000","last_clean":"2024-03-27T06:14:03.524898+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:03.524898+0000","last_undegraded":"2024-03-27T06:14:03.524898+0000","last_fullsized":"2024-03-27T06:14:03.524898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548043+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:03.548043+0000","last_peered":"2024-03-27T06:14:03.548043+0000","last_clean":"2024-03-27T06:14:03.548043+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:03.548043+0000","last_undegraded":"2024-03-27T06:14:03.548043+0000","last_fullsized":"2024-03-27T06:14:03.548043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.542134+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:03.542134+0000","last_peered":"2024-03-27T06:14:03.542134+0000","last_clean":"2024-03-27T06:14:03.542134+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:03.542134+0000","last_undegraded":"2024-03-27T06:14:03.542134+0000","last_fullsized":"2024-03-27T06:14:03.542134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546675+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:03.546675+0000","last_peered":"2024-03-27T06:14:03.546675+0000","last_clean":"2024-03-27T06:14:03.546675+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:03.546675+0000","last_undegraded":"2024-03-27T06:14:03.546675+0000","last_fullsized":"2024-03-27T06:14:03.546675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527566+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:04.527566+0000","last_peered":"2024-03-27T06:14:04.527566+0000","last_clean":"2024-03-27T06:14:04.527566+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:04.527566+0000","last_undegraded":"2024-03-27T06:14:04.527566+0000","last_fullsized":"2024-03-27T06:14:04.527566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.527486+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:03.527486+0000","last_peered":"2024-03-27T06:14:03.527486+0000","last_clean":"2024-03-27T06:14:03.527486+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:03.527486+0000","last_undegraded":"2024-03-27T06:14:03.527486+0000","last_fullsized":"2024-03-27T06:14:03.527486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557771+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:03.557771+0000","last_peered":"2024-03-27T06:14:03.557771+0000","last_clean":"2024-03-27T06:14:03.557771+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:03.557771+0000","last_undegraded":"2024-03-27T06:14:03.557771+0000","last_fullsized":"2024-03-27T06:14:03.557771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527630+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:04.527630+0000","last_peered":"2024-03-27T06:14:04.527630+0000","last_clean":"2024-03-27T06:14:04.527630+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:04.527630+0000","last_undegraded":"2024-03-27T06:14:04.527630+0000","last_fullsized":"2024-03-27T06:14:04.527630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541219+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:03.541219+0000","last_peered":"2024-03-27T06:14:03.541219+0000","last_clean":"2024-03-27T06:14:03.541219+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:03.541219+0000","last_undegraded":"2024-03-27T06:14:03.541219+0000","last_fullsized":"2024-03-27T06:14:03.541219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530841+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:04.530841+0000","last_peered":"2024-03-27T06:14:04.530841+0000","last_clean":"2024-03-27T06:14:04.530841+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:04.530841+0000","last_undegraded":"2024-03-27T06:14:04.530841+0000","last_fullsized":"2024-03-27T06:14:04.530841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536441+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:03.536441+0000","last_peered":"2024-03-27T06:14:03.536441+0000","last_clean":"2024-03-27T06:14:03.536441+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:03.536441+0000","last_undegraded":"2024-03-27T06:14:03.536441+0000","last_fullsized":"2024-03-27T06:14:03.536441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.539319+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:03.539319+0000","last_peered":"2024-03-27T06:14:03.539319+0000","last_clean":"2024-03-27T06:14:03.539319+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:03.539319+0000","last_undegraded":"2024-03-27T06:14:03.539319+0000","last_fullsized":"2024-03-27T06:14:03.539319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:42.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519128+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:03.519128+0000","last_peered":"2024-03-27T06:14:03.519128+0000","last_clean":"2024-03-27T06:14:03.519128+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:03.519128+0000","last_undegraded":"2024-03-27T06:14:03.519128+0000","last_fullsized":"2024-03-27T06:14:03.519128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546269+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:03.546269+0000","last_peered":"2024-03-27T06:14:03.546269+0000","last_clean":"2024-03-27T06:14:03.546269+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:03.546269+0000","last_undegraded":"2024-03-27T06:14:03.546269+0000","last_fullsized":"2024-03-27T06:14:03.546269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556577+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:03.556577+0000","last_peered":"2024-03-27T06:14:03.556577+0000","last_clean":"2024-03-27T06:14:03.556577+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:03.556577+0000","last_undegraded":"2024-03-27T06:14:03.556577+0000","last_fullsized":"2024-03-27T06:14:03.556577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.521455+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:03.521455+0000","last_peered":"2024-03-27T06:14:03.521455+0000","last_clean":"2024-03-27T06:14:03.521455+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:03.521455+0000","last_undegraded":"2024-03-27T06:14:03.521455+0000","last_fullsized":"2024-03-27T06:14:03.521455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561082+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:03.561082+0000","last_peered":"2024-03-27T06:14:03.561082+0000","last_clean":"2024-03-27T06:14:03.561082+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:03.561082+0000","last_undegraded":"2024-03-27T06:14:03.561082+0000","last_fullsized":"2024-03-27T06:14:03.561082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516009+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:03.516009+0000","last_peered":"2024-03-27T06:14:03.516009+0000","last_clean":"2024-03-27T06:14:03.516009+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:03.516009+0000","last_undegraded":"2024-03-27T06:14:03.516009+0000","last_fullsized":"2024-03-27T06:14:03.516009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556704+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:03.556704+0000","last_peered":"2024-03-27T06:14:03.556704+0000","last_clean":"2024-03-27T06:14:03.556704+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:03.556704+0000","last_undegraded":"2024-03-27T06:14:03.556704+0000","last_fullsized":"2024-03-27T06:14:03.556704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.513668+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:03.513668+0000","last_peered":"2024-03-27T06:14:03.513668+0000","last_clean":"2024-03-27T06:14:03.513668+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:03.513668+0000","last_undegraded":"2024-03-27T06:14:03.513668+0000","last_fullsized":"2024-03-27T06:14:03.513668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519312+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:03.519312+0000","last_peered":"2024-03-27T06:14:03.519312+0000","last_clean":"2024-03-27T06:14:03.519312+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:03.519312+0000","last_undegraded":"2024-03-27T06:14:03.519312+0000","last_fullsized":"2024-03-27T06:14:03.519312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531520+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:04.531520+0000","last_peered":"2024-03-27T06:14:04.531520+0000","last_clean":"2024-03-27T06:14:04.531520+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:04.531520+0000","last_undegraded":"2024-03-27T06:14:04.531520+0000","last_fullsized":"2024-03-27T06:14:04.531520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546488+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:03.546488+0000","last_peered":"2024-03-27T06:14:03.546488+0000","last_clean":"2024-03-27T06:14:03.546488+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:03.546488+0000","last_undegraded":"2024-03-27T06:14:03.546488+0000","last_fullsized":"2024-03-27T06:14:03.546488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529895+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:04.529895+0000","last_peered":"2024-03-27T06:14:04.529895+0000","last_clean":"2024-03-27T06:14:04.529895+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:04.529895+0000","last_undegraded":"2024-03-27T06:14:04.529895+0000","last_fullsized":"2024-03-27T06:14:04.529895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.529614+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:03.529614+0000","last_peered":"2024-03-27T06:14:03.529614+0000","last_clean":"2024-03-27T06:14:03.529614+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:03.529614+0000","last_undegraded":"2024-03-27T06:14:03.529614+0000","last_fullsized":"2024-03-27T06:14:03.529614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527370+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:04.527370+0000","last_peered":"2024-03-27T06:14:04.527370+0000","last_clean":"2024-03-27T06:14:04.527370+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:04.527370+0000","last_undegraded":"2024-03-27T06:14:04.527370+0000","last_fullsized":"2024-03-27T06:14:04.527370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.532114+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:03.532114+0000","last_peered":"2024-03-27T06:14:03.532114+0000","last_clean":"2024-03-27T06:14:03.532114+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:03.532114+0000","last_undegraded":"2024-03-27T06:14:03.532114+0000","last_fullsized":"2024-03-27T06:14:03.532114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531072+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:04.531072+0000","last_peered":"2024-03-27T06:14:04.531072+0000","last_clean":"2024-03-27T06:14:04.531072+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:04.531072+0000","last_undegraded":"2024-03-27T06:14:04.531072+0000","last_fullsized":"2024-03-27T06:14:04.531072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557762+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:03.557762+0000","last_peered":"2024-03-27T06:14:03.557762+0000","last_clean":"2024-03-27T06:14:03.557762+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:03.557762+0000","last_undegraded":"2024-03-27T06:14:03.557762+0000","last_fullsized":"2024-03-27T06:14:03.557762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516710+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:03.516710+0000","last_peered":"2024-03-27T06:14:03.516710+0000","last_clean":"2024-03-27T06:14:03.516710+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:03.516710+0000","last_undegraded":"2024-03-27T06:14:03.516710+0000","last_fullsized":"2024-03-27T06:14:03.516710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.528788+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:03.528788+0000","last_peered":"2024-03-27T06:14:03.528788+0000","last_clean":"2024-03-27T06:14:03.528788+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:03.528788+0000","last_undegraded":"2024-03-27T06:14:03.528788+0000","last_fullsized":"2024-03-27T06:14:03.528788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:33:24.757683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530500+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:04.530500+0000","last_peered":"2024-03-27T06:14:04.530500+0000","last_clean":"2024-03-27T06:14:04.530500+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:04.530500+0000","last_undegraded":"2024-03-27T06:14:04.530500+0000","last_fullsized":"2024-03-27T06:14:04.530500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523771+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:03.523771+0000","last_peered":"2024-03-27T06:14:03.523771+0000","last_clean":"2024-03-27T06:14:03.523771+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:03.523771+0000","last_undegraded":"2024-03-27T06:14:03.523771+0000","last_fullsized":"2024-03-27T06:14:03.523771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.521589+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:03.521589+0000","last_peered":"2024-03-27T06:14:03.521589+0000","last_clean":"2024-03-27T06:14:03.521589+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:03.521589+0000","last_undegraded":"2024-03-27T06:14:03.521589+0000","last_fullsized":"2024-03-27T06:14:03.521589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536268+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:03.536268+0000","last_peered":"2024-03-27T06:14:03.536268+0000","last_clean":"2024-03-27T06:14:03.536268+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:03.536268+0000","last_undegraded":"2024-03-27T06:14:03.536268+0000","last_fullsized":"2024-03-27T06:14:03.536268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530937+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:04.530937+0000","last_peered":"2024-03-27T06:14:04.530937+0000","last_clean":"2024-03-27T06:14:04.530937+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:04.530937+0000","last_undegraded":"2024-03-27T06:14:04.530937+0000","last_fullsized":"2024-03-27T06:14:04.530937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535865+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:03.535865+0000","last_peered":"2024-03-27T06:14:03.535865+0000","last_clean":"2024-03-27T06:14:03.535865+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:03.535865+0000","last_undegraded":"2024-03-27T06:14:03.535865+0000","last_fullsized":"2024-03-27T06:14:03.535865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.530644+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:03.530644+0000","last_peered":"2024-03-27T06:14:03.530644+0000","last_clean":"2024-03-27T06:14:03.530644+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:03.530644+0000","last_undegraded":"2024-03-27T06:14:03.530644+0000","last_fullsized":"2024-03-27T06:14:03.530644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.518673+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:03.518673+0000","last_peered":"2024-03-27T06:14:03.518673+0000","last_clean":"2024-03-27T06:14:03.518673+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:03.518673+0000","last_undegraded":"2024-03-27T06:14:03.518673+0000","last_fullsized":"2024-03-27T06:14:03.518673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.529213+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:03.529213+0000","last_peered":"2024-03-27T06:14:03.529213+0000","last_clean":"2024-03-27T06:14:03.529213+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:03.529213+0000","last_undegraded":"2024-03-27T06:14:03.529213+0000","last_fullsized":"2024-03-27T06:14:03.529213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.513144+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:03.513144+0000","last_peered":"2024-03-27T06:14:03.513144+0000","last_clean":"2024-03-27T06:14:03.513144+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:03.513144+0000","last_undegraded":"2024-03-27T06:14:03.513144+0000","last_fullsized":"2024-03-27T06:14:03.513144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531387+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:04.531387+0000","last_peered":"2024-03-27T06:14:04.531387+0000","last_clean":"2024-03-27T06:14:04.531387+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:04.531387+0000","last_undegraded":"2024-03-27T06:14:04.531387+0000","last_fullsized":"2024-03-27T06:14:04.531387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548126+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:03.548126+0000","last_peered":"2024-03-27T06:14:03.548126+0000","last_clean":"2024-03-27T06:14:03.548126+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:03.548126+0000","last_undegraded":"2024-03-27T06:14:03.548126+0000","last_fullsized":"2024-03-27T06:14:03.548126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.558729+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:03.558729+0000","last_peered":"2024-03-27T06:14:03.558729+0000","last_clean":"2024-03-27T06:14:03.558729+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:03.558729+0000","last_undegraded":"2024-03-27T06:14:03.558729+0000","last_fullsized":"2024-03-27T06:14:03.558729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541573+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:03.541573+0000","last_peered":"2024-03-27T06:14:03.541573+0000","last_clean":"2024-03-27T06:14:03.541573+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:03.541573+0000","last_undegraded":"2024-03-27T06:14:03.541573+0000","last_fullsized":"2024-03-27T06:14:03.541573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536189+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:03.536189+0000","last_peered":"2024-03-27T06:14:03.536189+0000","last_clean":"2024-03-27T06:14:03.536189+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:03.536189+0000","last_undegraded":"2024-03-27T06:14:03.536189+0000","last_fullsized":"2024-03-27T06:14:03.536189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519203+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:03.519203+0000","last_peered":"2024-03-27T06:14:03.519203+0000","last_clean":"2024-03-27T06:14:03.519203+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:03.519203+0000","last_undegraded":"2024-03-27T06:14:03.519203+0000","last_fullsized":"2024-03-27T06:14:03.519203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536220+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:03.536220+0000","last_peered":"2024-03-27T06:14:03.536220+0000","last_clean":"2024-03-27T06:14:03.536220+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:03.536220+0000","last_undegraded":"2024-03-27T06:14:03.536220+0000","last_fullsized":"2024-03-27T06:14:03.536220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.518223+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:03.518223+0000","last_peered":"2024-03-27T06:14:03.518223+0000","last_clean":"2024-03-27T06:14:03.518223+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:03.518223+0000","last_undegraded":"2024-03-27T06:14:03.518223+0000","last_fullsized":"2024-03-27T06:14:03.518223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:09.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:10.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.212+0000 7f85fd000640 1 Processor -- start 2024-03-27T06:14:10.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.212+0000 7f85fd000640 1 -- start start 2024-03-27T06:14:10.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85fd000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f85f806b520 0x7f85f806b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85fd000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f815b790 con 0x7f85f815aa00 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85fd000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f80c12d0 con 0x7f85f80b11e0 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85fd000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f85f80c1080 con 0x7f85f806b520 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f85f806b520 0x7f85f806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f85f806b520 0x7f85f806b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56776/0 (socket says 172.21.15.132:56776) 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 -- 172.21.15.132:0/711809744 learned_addr learned my addr 172.21.15.132:0/711809744 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 -- 172.21.15.132:0/711809744 >> v1:172.21.15.134:6789/0 conn(0x7f85f80b11e0 legacy=0x7f85f80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 -- 172.21.15.132:0/711809744 >> v1:172.21.15.132:6789/0 conn(0x7f85f815aa00 legacy=0x7f85f815ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.214+0000 7f85f5400640 1 -- 172.21.15.132:0/711809744 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f80c15b0 con 0x7f85f806b520 2024-03-27T06:14:10.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.215+0000 7f85f5400640 1 --2- 172.21.15.132:0/711809744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f806b520 0x7f85f806b960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009870 tx=0x7f85e402bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985db673ff76815d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.216+0000 7f85f4a00640 1 -- 172.21.15.132:0/711809744 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e402d870 con 0x7f85f806b520 2024-03-27T06:14:10.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.216+0000 7f85f4a00640 1 -- 172.21.15.132:0/711809744 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e402da10 con 0x7f85f806b520 2024-03-27T06:14:10.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.216+0000 7f85f4a00640 1 -- 172.21.15.132:0/711809744 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e402dde0 con 0x7f85f806b520 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.216+0000 7f85fd000640 1 -- 172.21.15.132:0/711809744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f806b520 msgr2=0x7f85f806b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.216+0000 7f85fd000640 1 --2- 172.21.15.132:0/711809744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f806b520 0x7f85f806b960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009870 tx=0x7f85e402bab0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.217+0000 7f85fd000640 1 -- 172.21.15.132:0/711809744 shutdown_connections 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.217+0000 7f85fd000640 1 --2- 172.21.15.132:0/711809744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f806b520 0x7f85f806b960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.217+0000 7f85fd000640 1 -- 172.21.15.132:0/711809744 >> 172.21.15.132:0/711809744 conn(0x7f85f80cc060 msgr2=0x7f85f80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.217+0000 7f85fd000640 1 -- 172.21.15.132:0/711809744 shutdown_connections 2024-03-27T06:14:10.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.217+0000 7f85fd000640 1 -- 172.21.15.132:0/711809744 wait complete. 2024-03-27T06:14:10.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.219+0000 7f85fd000640 1 Processor -- start 2024-03-27T06:14:10.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.219+0000 7f85fd000640 1 -- start start 2024-03-27T06:14:10.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f815aa00 0x7f85f80c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85f80b15c0 con 0x7f85f815aa00 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f85f80c1c00 con 0x7f85f806b520 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85fd000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f85f80c2090 con 0x7f85f80b11e0 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f6800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f815aa00 0x7f85f80c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:41150/0 (socket says 172.21.15.132:41150) 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:60752/0 (socket says 172.21.15.132:60752) 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.220+0000 7f85f5400640 1 -- 172.21.15.132:0/4183232239 learned_addr learned my addr 172.21.15.132:0/4183232239 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:10.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 -- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 msgr2=0x7f85f81f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 -- 172.21.15.132:0/4183232239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f815aa00 msgr2=0x7f85f80c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f815aa00 0x7f85f80c3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 -- 172.21.15.132:0/4183232239 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f80c52c0 con 0x7f85f80b11e0 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5e00640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:10.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85f5400640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009990 tx=0x7f85e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4047070 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e4037a40 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4037e10 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.221+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f80c5580 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.222+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f80c5b40 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.222+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f8068840 con 0x7f85f80b11e0 2024-03-27T06:14:10.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.224+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f85e404b430 con 0x7f85f80b11e0 2024-03-27T06:14:10.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.224+0000 7f85ee000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 0x7f85d806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.225+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f85e40ad270 con 0x7f85f80b11e0 2024-03-27T06:14:10.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.225+0000 7f85f5e00640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 0x7f85d806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.225+0000 7f85f5e00640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 0x7f85d806e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85e8000f60 tx=0x7f85e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.230+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f85e407b1a0 con 0x7f85f80b11e0 2024-03-27T06:14:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.427+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f85f80737e0 con 0x7f85d806c530 2024-03-27T06:14:10.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.434+0000 7f85ee000640 1 -- 172.21.15.132:0/4183232239 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251871 (secure 0 0 0) 0x7f85f80737e0 con 0x7f85d806c530 2024-03-27T06:14:10.428 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:10.430 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 msgr2=0x7f85d806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 0x7f85d806e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85e8000f60 tx=0x7f85e8009040 comp rx=0 tx=0).stop 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 msgr2=0x7f85f8151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009990 tx=0x7f85e4004730 comp rx=0 tx=0).stop 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 shutdown_connections 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85f815aa00 0x7f85f80c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85f80b11e0 0x7f85f8151c00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85d806c530 0x7f85d806e9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 --2- 172.21.15.132:0/4183232239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85f806b520 0x7f85f81f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.440+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 >> 172.21.15.132:0/4183232239 conn(0x7f85f80cc060 msgr2=0x7f85f8153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:10.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.441+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 shutdown_connections 2024-03-27T06:14:10.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.441+0000 7f85fd000640 1 -- 172.21.15.132:0/4183232239 wait complete. 2024-03-27T06:14:10.449 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-27T06:14:10.392612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217960,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756760,"statfs":{"total":773094113280,"available":772870922240,"internally_reserved":0,"allocated":1794048,"data_stored":1747776,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.888190"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561103+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:03.561103+0000","last_peered":"2024-03-27T06:14:03.561103+0000","last_clean":"2024-03-27T06:14:03.561103+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:03.561103+0000","last_undegraded":"2024-03-27T06:14:03.561103+0000","last_fullsized":"2024-03-27T06:14:03.561103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535967+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:03.535967+0000","last_peered":"2024-03-27T06:14:03.535967+0000","last_clean":"2024-03-27T06:14:03.535967+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:03.535967+0000","last_undegraded":"2024-03-27T06:14:03.535967+0000","last_fullsized":"2024-03-27T06:14:03.535967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536186+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:03.536186+0000","last_peered":"2024-03-27T06:14:03.536186+0000","last_clean":"2024-03-27T06:14:03.536186+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:03.536186+0000","last_undegraded":"2024-03-27T06:14:03.536186+0000","last_fullsized":"2024-03-27T06:14:03.536186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.538428+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:03.538428+0000","last_peered":"2024-03-27T06:14:03.538428+0000","last_clean":"2024-03-27T06:14:03.538428+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:03.538428+0000","last_undegraded":"2024-03-27T06:14:03.538428+0000","last_fullsized":"2024-03-27T06:14:03.538428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:53.372177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556789+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:03.556789+0000","last_peered":"2024-03-27T06:14:03.556789+0000","last_clean":"2024-03-27T06:14:03.556789+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:03.556789+0000","last_undegraded":"2024-03-27T06:14:03.556789+0000","last_fullsized":"2024-03-27T06:14:03.556789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.543307+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:03.543307+0000","last_peered":"2024-03-27T06:14:03.543307+0000","last_clean":"2024-03-27T06:14:03.543307+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:03.543307+0000","last_undegraded":"2024-03-27T06:14:03.543307+0000","last_fullsized":"2024-03-27T06:14:03.543307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:42.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552053+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:03.552053+0000","last_peered":"2024-03-27T06:14:03.552053+0000","last_clean":"2024-03-27T06:14:03.552053+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:03.552053+0000","last_undegraded":"2024-03-27T06:14:03.552053+0000","last_fullsized":"2024-03-27T06:14:03.552053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546650+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:03.546650+0000","last_peered":"2024-03-27T06:14:03.546650+0000","last_clean":"2024-03-27T06:14:03.546650+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:03.546650+0000","last_undegraded":"2024-03-27T06:14:03.546650+0000","last_fullsized":"2024-03-27T06:14:03.546650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.545757+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:03.545757+0000","last_peered":"2024-03-27T06:14:03.545757+0000","last_clean":"2024-03-27T06:14:03.545757+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:03.545757+0000","last_undegraded":"2024-03-27T06:14:03.545757+0000","last_fullsized":"2024-03-27T06:14:03.545757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.558252+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:03.558252+0000","last_peered":"2024-03-27T06:14:03.558252+0000","last_clean":"2024-03-27T06:14:03.558252+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:03.558252+0000","last_undegraded":"2024-03-27T06:14:03.558252+0000","last_fullsized":"2024-03-27T06:14:03.558252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557622+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:03.557622+0000","last_peered":"2024-03-27T06:14:03.557622+0000","last_clean":"2024-03-27T06:14:03.557622+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:03.557622+0000","last_undegraded":"2024-03-27T06:14:03.557622+0000","last_fullsized":"2024-03-27T06:14:03.557622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.559083+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:03.559083+0000","last_peered":"2024-03-27T06:14:03.559083+0000","last_clean":"2024-03-27T06:14:03.559083+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:03.559083+0000","last_undegraded":"2024-03-27T06:14:03.559083+0000","last_fullsized":"2024-03-27T06:14:03.559083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.554961+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:03.554961+0000","last_peered":"2024-03-27T06:14:03.554961+0000","last_clean":"2024-03-27T06:14:03.554961+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:03.554961+0000","last_undegraded":"2024-03-27T06:14:03.554961+0000","last_fullsized":"2024-03-27T06:14:03.554961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523526+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:03.523526+0000","last_peered":"2024-03-27T06:14:03.523526+0000","last_clean":"2024-03-27T06:14:03.523526+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:03.523526+0000","last_undegraded":"2024-03-27T06:14:03.523526+0000","last_fullsized":"2024-03-27T06:14:03.523526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552785+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:03.552785+0000","last_peered":"2024-03-27T06:14:03.552785+0000","last_clean":"2024-03-27T06:14:03.552785+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:03.552785+0000","last_undegraded":"2024-03-27T06:14:03.552785+0000","last_fullsized":"2024-03-27T06:14:03.552785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556728+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:03.556728+0000","last_peered":"2024-03-27T06:14:03.556728+0000","last_clean":"2024-03-27T06:14:03.556728+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:03.556728+0000","last_undegraded":"2024-03-27T06:14:03.556728+0000","last_fullsized":"2024-03-27T06:14:03.556728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535810+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:03.535810+0000","last_peered":"2024-03-27T06:14:03.535810+0000","last_clean":"2024-03-27T06:14:03.535810+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:03.535810+0000","last_undegraded":"2024-03-27T06:14:03.535810+0000","last_fullsized":"2024-03-27T06:14:03.535810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561781+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:03.561781+0000","last_peered":"2024-03-27T06:14:03.561781+0000","last_clean":"2024-03-27T06:14:03.561781+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:03.561781+0000","last_undegraded":"2024-03-27T06:14:03.561781+0000","last_fullsized":"2024-03-27T06:14:03.561781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536210+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:03.536210+0000","last_peered":"2024-03-27T06:14:03.536210+0000","last_clean":"2024-03-27T06:14:03.536210+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:03.536210+0000","last_undegraded":"2024-03-27T06:14:03.536210+0000","last_fullsized":"2024-03-27T06:14:03.536210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541584+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:03.541584+0000","last_peered":"2024-03-27T06:14:03.541584+0000","last_clean":"2024-03-27T06:14:03.541584+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:03.541584+0000","last_undegraded":"2024-03-27T06:14:03.541584+0000","last_fullsized":"2024-03-27T06:14:03.541584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548022+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:03.548022+0000","last_peered":"2024-03-27T06:14:03.548022+0000","last_clean":"2024-03-27T06:14:03.548022+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:03.548022+0000","last_undegraded":"2024-03-27T06:14:03.548022+0000","last_fullsized":"2024-03-27T06:14:03.548022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523683+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:03.523683+0000","last_peered":"2024-03-27T06:14:03.523683+0000","last_clean":"2024-03-27T06:14:03.523683+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:03.523683+0000","last_undegraded":"2024-03-27T06:14:03.523683+0000","last_fullsized":"2024-03-27T06:14:03.523683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:38.647261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.562803+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:03.562803+0000","last_peered":"2024-03-27T06:14:03.562803+0000","last_clean":"2024-03-27T06:14:03.562803+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:03.562803+0000","last_undegraded":"2024-03-27T06:14:03.562803+0000","last_fullsized":"2024-03-27T06:14:03.562803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536030+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:03.536030+0000","last_peered":"2024-03-27T06:14:03.536030+0000","last_clean":"2024-03-27T06:14:03.536030+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:03.536030+0000","last_undegraded":"2024-03-27T06:14:03.536030+0000","last_fullsized":"2024-03-27T06:14:03.536030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.530720+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:03.530720+0000","last_peered":"2024-03-27T06:14:03.530720+0000","last_clean":"2024-03-27T06:14:03.530720+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:03.530720+0000","last_undegraded":"2024-03-27T06:14:03.530720+0000","last_fullsized":"2024-03-27T06:14:03.530720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:08:35.447018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536752+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:03.536752+0000","last_peered":"2024-03-27T06:14:03.536752+0000","last_clean":"2024-03-27T06:14:03.536752+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:03.536752+0000","last_undegraded":"2024-03-27T06:14:03.536752+0000","last_fullsized":"2024-03-27T06:14:03.536752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:53.372177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516878+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:03.516878+0000","last_peered":"2024-03-27T06:14:03.516878+0000","last_clean":"2024-03-27T06:14:03.516878+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:03.516878+0000","last_undegraded":"2024-03-27T06:14:03.516878+0000","last_fullsized":"2024-03-27T06:14:03.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.552275+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:03.552275+0000","last_peered":"2024-03-27T06:14:03.552275+0000","last_clean":"2024-03-27T06:14:03.552275+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:03.552275+0000","last_undegraded":"2024-03-27T06:14:03.552275+0000","last_fullsized":"2024-03-27T06:14:03.552275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530565+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:04.530565+0000","last_peered":"2024-03-27T06:14:04.530565+0000","last_clean":"2024-03-27T06:14:04.530565+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:04.530565+0000","last_undegraded":"2024-03-27T06:14:04.530565+0000","last_fullsized":"2024-03-27T06:14:04.530565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.512599+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:03.512599+0000","last_peered":"2024-03-27T06:14:03.512599+0000","last_clean":"2024-03-27T06:14:03.512599+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:03.512599+0000","last_undegraded":"2024-03-27T06:14:03.512599+0000","last_fullsized":"2024-03-27T06:14:03.512599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536047+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:03.536047+0000","last_peered":"2024-03-27T06:14:03.536047+0000","last_clean":"2024-03-27T06:14:03.536047+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:03.536047+0000","last_undegraded":"2024-03-27T06:14:03.536047+0000","last_fullsized":"2024-03-27T06:14:03.536047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529878+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:04.529878+0000","last_peered":"2024-03-27T06:14:04.529878+0000","last_clean":"2024-03-27T06:14:04.529878+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:04.529878+0000","last_undegraded":"2024-03-27T06:14:04.529878+0000","last_fullsized":"2024-03-27T06:14:04.529878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536255+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:03.536255+0000","last_peered":"2024-03-27T06:14:03.536255+0000","last_clean":"2024-03-27T06:14:03.536255+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:03.536255+0000","last_undegraded":"2024-03-27T06:14:03.536255+0000","last_fullsized":"2024-03-27T06:14:03.536255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.524898+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:03.524898+0000","last_peered":"2024-03-27T06:14:03.524898+0000","last_clean":"2024-03-27T06:14:03.524898+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:03.524898+0000","last_undegraded":"2024-03-27T06:14:03.524898+0000","last_fullsized":"2024-03-27T06:14:03.524898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548043+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:03.548043+0000","last_peered":"2024-03-27T06:14:03.548043+0000","last_clean":"2024-03-27T06:14:03.548043+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:03.548043+0000","last_undegraded":"2024-03-27T06:14:03.548043+0000","last_fullsized":"2024-03-27T06:14:03.548043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.542134+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:03.542134+0000","last_peered":"2024-03-27T06:14:03.542134+0000","last_clean":"2024-03-27T06:14:03.542134+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:03.542134+0000","last_undegraded":"2024-03-27T06:14:03.542134+0000","last_fullsized":"2024-03-27T06:14:03.542134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546675+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:03.546675+0000","last_peered":"2024-03-27T06:14:03.546675+0000","last_clean":"2024-03-27T06:14:03.546675+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:03.546675+0000","last_undegraded":"2024-03-27T06:14:03.546675+0000","last_fullsized":"2024-03-27T06:14:03.546675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527566+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:04.527566+0000","last_peered":"2024-03-27T06:14:04.527566+0000","last_clean":"2024-03-27T06:14:04.527566+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:04.527566+0000","last_undegraded":"2024-03-27T06:14:04.527566+0000","last_fullsized":"2024-03-27T06:14:04.527566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.527486+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:03.527486+0000","last_peered":"2024-03-27T06:14:03.527486+0000","last_clean":"2024-03-27T06:14:03.527486+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:03.527486+0000","last_undegraded":"2024-03-27T06:14:03.527486+0000","last_fullsized":"2024-03-27T06:14:03.527486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557771+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:03.557771+0000","last_peered":"2024-03-27T06:14:03.557771+0000","last_clean":"2024-03-27T06:14:03.557771+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:03.557771+0000","last_undegraded":"2024-03-27T06:14:03.557771+0000","last_fullsized":"2024-03-27T06:14:03.557771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527630+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:04.527630+0000","last_peered":"2024-03-27T06:14:04.527630+0000","last_clean":"2024-03-27T06:14:04.527630+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:04.527630+0000","last_undegraded":"2024-03-27T06:14:04.527630+0000","last_fullsized":"2024-03-27T06:14:04.527630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541219+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:03.541219+0000","last_peered":"2024-03-27T06:14:03.541219+0000","last_clean":"2024-03-27T06:14:03.541219+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:03.541219+0000","last_undegraded":"2024-03-27T06:14:03.541219+0000","last_fullsized":"2024-03-27T06:14:03.541219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530841+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:04.530841+0000","last_peered":"2024-03-27T06:14:04.530841+0000","last_clean":"2024-03-27T06:14:04.530841+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:04.530841+0000","last_undegraded":"2024-03-27T06:14:04.530841+0000","last_fullsized":"2024-03-27T06:14:04.530841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536441+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:03.536441+0000","last_peered":"2024-03-27T06:14:03.536441+0000","last_clean":"2024-03-27T06:14:03.536441+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:03.536441+0000","last_undegraded":"2024-03-27T06:14:03.536441+0000","last_fullsized":"2024-03-27T06:14:03.536441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.539319+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:03.539319+0000","last_peered":"2024-03-27T06:14:03.539319+0000","last_clean":"2024-03-27T06:14:03.539319+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:03.539319+0000","last_undegraded":"2024-03-27T06:14:03.539319+0000","last_fullsized":"2024-03-27T06:14:03.539319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:42.320226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519128+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:03.519128+0000","last_peered":"2024-03-27T06:14:03.519128+0000","last_clean":"2024-03-27T06:14:03.519128+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:03.519128+0000","last_undegraded":"2024-03-27T06:14:03.519128+0000","last_fullsized":"2024-03-27T06:14:03.519128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546269+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:03.546269+0000","last_peered":"2024-03-27T06:14:03.546269+0000","last_clean":"2024-03-27T06:14:03.546269+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:03.546269+0000","last_undegraded":"2024-03-27T06:14:03.546269+0000","last_fullsized":"2024-03-27T06:14:03.546269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556577+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:03.556577+0000","last_peered":"2024-03-27T06:14:03.556577+0000","last_clean":"2024-03-27T06:14:03.556577+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:03.556577+0000","last_undegraded":"2024-03-27T06:14:03.556577+0000","last_fullsized":"2024-03-27T06:14:03.556577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.521455+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:03.521455+0000","last_peered":"2024-03-27T06:14:03.521455+0000","last_clean":"2024-03-27T06:14:03.521455+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:03.521455+0000","last_undegraded":"2024-03-27T06:14:03.521455+0000","last_fullsized":"2024-03-27T06:14:03.521455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.561082+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:03.561082+0000","last_peered":"2024-03-27T06:14:03.561082+0000","last_clean":"2024-03-27T06:14:03.561082+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:03.561082+0000","last_undegraded":"2024-03-27T06:14:03.561082+0000","last_fullsized":"2024-03-27T06:14:03.561082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516009+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:03.516009+0000","last_peered":"2024-03-27T06:14:03.516009+0000","last_clean":"2024-03-27T06:14:03.516009+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:03.516009+0000","last_undegraded":"2024-03-27T06:14:03.516009+0000","last_fullsized":"2024-03-27T06:14:03.516009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.556704+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:03.556704+0000","last_peered":"2024-03-27T06:14:03.556704+0000","last_clean":"2024-03-27T06:14:03.556704+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:03.556704+0000","last_undegraded":"2024-03-27T06:14:03.556704+0000","last_fullsized":"2024-03-27T06:14:03.556704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.513668+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:03.513668+0000","last_peered":"2024-03-27T06:14:03.513668+0000","last_clean":"2024-03-27T06:14:03.513668+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:03.513668+0000","last_undegraded":"2024-03-27T06:14:03.513668+0000","last_fullsized":"2024-03-27T06:14:03.513668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519312+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:03.519312+0000","last_peered":"2024-03-27T06:14:03.519312+0000","last_clean":"2024-03-27T06:14:03.519312+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:03.519312+0000","last_undegraded":"2024-03-27T06:14:03.519312+0000","last_fullsized":"2024-03-27T06:14:03.519312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531520+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:04.531520+0000","last_peered":"2024-03-27T06:14:04.531520+0000","last_clean":"2024-03-27T06:14:04.531520+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:04.531520+0000","last_undegraded":"2024-03-27T06:14:04.531520+0000","last_fullsized":"2024-03-27T06:14:04.531520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.546488+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:03.546488+0000","last_peered":"2024-03-27T06:14:03.546488+0000","last_clean":"2024-03-27T06:14:03.546488+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:03.546488+0000","last_undegraded":"2024-03-27T06:14:03.546488+0000","last_fullsized":"2024-03-27T06:14:03.546488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529895+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:04.529895+0000","last_peered":"2024-03-27T06:14:04.529895+0000","last_clean":"2024-03-27T06:14:04.529895+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:04.529895+0000","last_undegraded":"2024-03-27T06:14:04.529895+0000","last_fullsized":"2024-03-27T06:14:04.529895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.529614+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:03.529614+0000","last_peered":"2024-03-27T06:14:03.529614+0000","last_clean":"2024-03-27T06:14:03.529614+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:03.529614+0000","last_undegraded":"2024-03-27T06:14:03.529614+0000","last_fullsized":"2024-03-27T06:14:03.529614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527370+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:04.527370+0000","last_peered":"2024-03-27T06:14:04.527370+0000","last_clean":"2024-03-27T06:14:04.527370+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:04.527370+0000","last_undegraded":"2024-03-27T06:14:04.527370+0000","last_fullsized":"2024-03-27T06:14:04.527370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.532114+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:03.532114+0000","last_peered":"2024-03-27T06:14:03.532114+0000","last_clean":"2024-03-27T06:14:03.532114+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:03.532114+0000","last_undegraded":"2024-03-27T06:14:03.532114+0000","last_fullsized":"2024-03-27T06:14:03.532114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531072+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:04.531072+0000","last_peered":"2024-03-27T06:14:04.531072+0000","last_clean":"2024-03-27T06:14:04.531072+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:04.531072+0000","last_undegraded":"2024-03-27T06:14:04.531072+0000","last_fullsized":"2024-03-27T06:14:04.531072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.557762+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:03.557762+0000","last_peered":"2024-03-27T06:14:03.557762+0000","last_clean":"2024-03-27T06:14:03.557762+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:03.557762+0000","last_undegraded":"2024-03-27T06:14:03.557762+0000","last_fullsized":"2024-03-27T06:14:03.557762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.516710+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:03.516710+0000","last_peered":"2024-03-27T06:14:03.516710+0000","last_clean":"2024-03-27T06:14:03.516710+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:03.516710+0000","last_undegraded":"2024-03-27T06:14:03.516710+0000","last_fullsized":"2024-03-27T06:14:03.516710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.528788+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:03.528788+0000","last_peered":"2024-03-27T06:14:03.528788+0000","last_clean":"2024-03-27T06:14:03.528788+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:03.528788+0000","last_undegraded":"2024-03-27T06:14:03.528788+0000","last_fullsized":"2024-03-27T06:14:03.528788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:33:24.757683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530500+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:04.530500+0000","last_peered":"2024-03-27T06:14:04.530500+0000","last_clean":"2024-03-27T06:14:04.530500+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:04.530500+0000","last_undegraded":"2024-03-27T06:14:04.530500+0000","last_fullsized":"2024-03-27T06:14:04.530500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.523771+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:03.523771+0000","last_peered":"2024-03-27T06:14:03.523771+0000","last_clean":"2024-03-27T06:14:03.523771+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:03.523771+0000","last_undegraded":"2024-03-27T06:14:03.523771+0000","last_fullsized":"2024-03-27T06:14:03.523771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:55.668439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.521589+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:03.521589+0000","last_peered":"2024-03-27T06:14:03.521589+0000","last_clean":"2024-03-27T06:14:03.521589+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:03.521589+0000","last_undegraded":"2024-03-27T06:14:03.521589+0000","last_fullsized":"2024-03-27T06:14:03.521589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536268+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:03.536268+0000","last_peered":"2024-03-27T06:14:03.536268+0000","last_clean":"2024-03-27T06:14:03.536268+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:03.536268+0000","last_undegraded":"2024-03-27T06:14:03.536268+0000","last_fullsized":"2024-03-27T06:14:03.536268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530937+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:04.530937+0000","last_peered":"2024-03-27T06:14:04.530937+0000","last_clean":"2024-03-27T06:14:04.530937+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:04.530937+0000","last_undegraded":"2024-03-27T06:14:04.530937+0000","last_fullsized":"2024-03-27T06:14:04.530937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.535865+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:03.535865+0000","last_peered":"2024-03-27T06:14:03.535865+0000","last_clean":"2024-03-27T06:14:03.535865+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:03.535865+0000","last_undegraded":"2024-03-27T06:14:03.535865+0000","last_fullsized":"2024-03-27T06:14:03.535865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.530644+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:03.530644+0000","last_peered":"2024-03-27T06:14:03.530644+0000","last_clean":"2024-03-27T06:14:03.530644+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:03.530644+0000","last_undegraded":"2024-03-27T06:14:03.530644+0000","last_fullsized":"2024-03-27T06:14:03.530644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.518673+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:03.518673+0000","last_peered":"2024-03-27T06:14:03.518673+0000","last_clean":"2024-03-27T06:14:03.518673+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:03.518673+0000","last_undegraded":"2024-03-27T06:14:03.518673+0000","last_fullsized":"2024-03-27T06:14:03.518673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.529213+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:03.529213+0000","last_peered":"2024-03-27T06:14:03.529213+0000","last_clean":"2024-03-27T06:14:03.529213+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:03.529213+0000","last_undegraded":"2024-03-27T06:14:03.529213+0000","last_fullsized":"2024-03-27T06:14:03.529213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.513144+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:03.513144+0000","last_peered":"2024-03-27T06:14:03.513144+0000","last_clean":"2024-03-27T06:14:03.513144+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:03.513144+0000","last_undegraded":"2024-03-27T06:14:03.513144+0000","last_fullsized":"2024-03-27T06:14:03.513144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531387+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:04.531387+0000","last_peered":"2024-03-27T06:14:04.531387+0000","last_clean":"2024-03-27T06:14:04.531387+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:04.531387+0000","last_undegraded":"2024-03-27T06:14:04.531387+0000","last_fullsized":"2024-03-27T06:14:04.531387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.548126+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:03.548126+0000","last_peered":"2024-03-27T06:14:03.548126+0000","last_clean":"2024-03-27T06:14:03.548126+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:03.548126+0000","last_undegraded":"2024-03-27T06:14:03.548126+0000","last_fullsized":"2024-03-27T06:14:03.548126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.558729+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:03.558729+0000","last_peered":"2024-03-27T06:14:03.558729+0000","last_clean":"2024-03-27T06:14:03.558729+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:03.558729+0000","last_undegraded":"2024-03-27T06:14:03.558729+0000","last_fullsized":"2024-03-27T06:14:03.558729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:15.662767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.541573+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:03.541573+0000","last_peered":"2024-03-27T06:14:03.541573+0000","last_clean":"2024-03-27T06:14:03.541573+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:03.541573+0000","last_undegraded":"2024-03-27T06:14:03.541573+0000","last_fullsized":"2024-03-27T06:14:03.541573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536189+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:03.536189+0000","last_peered":"2024-03-27T06:14:03.536189+0000","last_clean":"2024-03-27T06:14:03.536189+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:03.536189+0000","last_undegraded":"2024-03-27T06:14:03.536189+0000","last_fullsized":"2024-03-27T06:14:03.536189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.519203+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:03.519203+0000","last_peered":"2024-03-27T06:14:03.519203+0000","last_clean":"2024-03-27T06:14:03.519203+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:03.519203+0000","last_undegraded":"2024-03-27T06:14:03.519203+0000","last_fullsized":"2024-03-27T06:14:03.519203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.536220+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:03.536220+0000","last_peered":"2024-03-27T06:14:03.536220+0000","last_clean":"2024-03-27T06:14:03.536220+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:03.536220+0000","last_undegraded":"2024-03-27T06:14:03.536220+0000","last_fullsized":"2024-03-27T06:14:03.536220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T06:14:03.518223+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:03.518223+0000","last_peered":"2024-03-27T06:14:03.518223+0000","last_clean":"2024-03-27T06:14:03.518223+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:03.518223+0000","last_undegraded":"2024-03-27T06:14:03.518223+0000","last_fullsized":"2024-03-27T06:14:03.518223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:10.453 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-27T06:14:10.453 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-27T06:14:10.453 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-27T06:14:10.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-27T06:14:10.454 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-27T06:14:10.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.709+0000 7f3f8a600640 1 Processor -- start 2024-03-27T06:14:10.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.709+0000 7f3f8a600640 1 -- start start 2024-03-27T06:14:10.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.710+0000 7f3f8a600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3f84152950 0x7f3f841590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.710+0000 7f3f8a600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8406b630 con 0x7f3f84073970 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f8a600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8406b8d0 con 0x7f3f84073550 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f8a600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3f84152d30 con 0x7f3f84152950 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3f84152950 0x7f3f841590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f3f84073970 0x7f3f8406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42748/0 (socket says 172.21.15.132:42748) 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83e00640 1 -- 172.21.15.132:0/1194790588 learned_addr learned my addr 172.21.15.132:0/1194790588 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:10.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f82a00640 1 --1- 172.21.15.132:0/1194790588 >> v1:172.21.15.134:6789/0 conn(0x7f3f84073550 0x7f3f84071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55488/0 (socket says 172.21.15.132:55488) 2024-03-27T06:14:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83400640 1 -- 172.21.15.132:0/1194790588 >> v1:172.21.15.134:6789/0 conn(0x7f3f84073550 legacy=0x7f3f84071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83400640 1 -- 172.21.15.132:0/1194790588 >> v1:172.21.15.132:6789/0 conn(0x7f3f84073970 legacy=0x7f3f8406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.711+0000 7f3f83400640 1 -- 172.21.15.132:0/1194790588 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8406bbb0 con 0x7f3f84152950 2024-03-27T06:14:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.712+0000 7f3f83400640 1 --2- 172.21.15.132:0/1194790588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f841590a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097d0 tx=0x7f3f7402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acb842dd377d293d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f82000640 1 -- 172.21.15.132:0/1194790588 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74034070 con 0x7f3f84152950 2024-03-27T06:14:10.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f82000640 1 -- 172.21.15.132:0/1194790588 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f74030610 con 0x7f3f84152950 2024-03-27T06:14:10.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f82000640 1 -- 172.21.15.132:0/1194790588 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f740309e0 con 0x7f3f84152950 2024-03-27T06:14:10.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 -- 172.21.15.132:0/1194790588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 msgr2=0x7f3f841590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 --2- 172.21.15.132:0/1194790588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f841590a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097d0 tx=0x7f3f7402b960 comp rx=0 tx=0).stop 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 -- 172.21.15.132:0/1194790588 shutdown_connections 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 --2- 172.21.15.132:0/1194790588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f841590a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 -- 172.21.15.132:0/1194790588 >> 172.21.15.132:0/1194790588 conn(0x7f3f840cc060 msgr2=0x7f3f840cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.713+0000 7f3f8a600640 1 -- 172.21.15.132:0/1194790588 shutdown_connections 2024-03-27T06:14:10.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.714+0000 7f3f8a600640 1 -- 172.21.15.132:0/1194790588 wait complete. 2024-03-27T06:14:10.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.716+0000 7f3f8a600640 1 Processor -- start 2024-03-27T06:14:10.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.716+0000 7f3f8a600640 1 -- start start 2024-03-27T06:14:10.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3f84073970 0x7f3f841f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f8414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8406c450 con 0x7f3f84073550 2024-03-27T06:14:10.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8406c170 con 0x7f3f84152950 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f8a600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3f84071880 con 0x7f3f84073970 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f83400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f83e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f8414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f83400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56800/0 (socket says 172.21.15.132:56800) 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f82a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3f84073970 0x7f3f841f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.717+0000 7f3f83400640 1 -- 172.21.15.132:0/3581361210 learned_addr learned my addr 172.21.15.132:0/3581361210 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 -- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3f84073970 msgr2=0x7f3f841f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3f84073970 0x7f3f841f0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 -- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 msgr2=0x7f3f8414a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f8414a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 -- 172.21.15.132:0/3581361210 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f841f4880 con 0x7f3f84073550 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83e00640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f8414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f83400640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f740098f0 tx=0x7f3f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74034040 con 0x7f3f84073550 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f7403acc0 con 0x7f3f84073550 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74044630 con 0x7f3f84073550 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.718+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f840c1730 con 0x7f3f84073550 2024-03-27T06:14:10.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.719+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f841f3da0 con 0x7f3f84073550 2024-03-27T06:14:10.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.720+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f84068840 con 0x7f3f84073550 2024-03-27T06:14:10.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.720+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3f74037070 con 0x7f3f84073550 2024-03-27T06:14:10.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.721+0000 7f3f7b400640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 0x7f3f6406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:10.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.721+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3f740ad640 con 0x7f3f84073550 2024-03-27T06:14:10.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.722+0000 7f3f82a00640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 0x7f3f6406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:10.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.722+0000 7f3f82a00640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 0x7f3f6406e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3f840c9aa0 tx=0x7f3f70000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:10.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.726+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3f7403f070 con 0x7f3f84073550 2024-03-27T06:14:10.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.975+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3f840c6b30 con 0x7f3f84073550 2024-03-27T06:14:10.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.984+0000 7f3f7b400640 1 -- 172.21.15.132:0/3581361210 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173741 (secure 0 0 0) 0x7f3f7407b720 con 0x7f3f84073550 2024-03-27T06:14:10.978 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.988+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 msgr2=0x7f3f6406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.988+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 0x7f3f6406e9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3f840c9aa0 tx=0x7f3f70000ed0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.988+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 msgr2=0x7f3f840c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.988+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f740098f0 tx=0x7f3f740046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 shutdown_connections 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3f6406c530 0x7f3f6406e9f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f84152950 0x7f3f8414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3f84073970 0x7f3f841f0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 --2- 172.21.15.132:0/3581361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f84073550 0x7f3f840c1080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 >> 172.21.15.132:0/3581361210 conn(0x7f3f840cc060 msgr2=0x7f3f84071170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 shutdown_connections 2024-03-27T06:14:10.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:10.989+0000 7f3f8a600640 1 -- 172.21.15.132:0/3581361210 wait complete. 2024-03-27T06:14:10.997 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":2615516388},{"type":"v1","addr":"172.21.15.134:6833","nonce":2615516388}]},"active_addr":"172.21.15.134:6833/2615516388","active_change":"2024-03-27T06:13:48.378100+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1601862251}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1846574069}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":287206788}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3668969060}]}]} 2024-03-27T06:14:11.001 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-27T06:14:11.001 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T06:14:11.001 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.221+0000 7f8235a00640 1 Processor -- start 2024-03-27T06:14:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.221+0000 7f8235a00640 1 -- start start 2024-03-27T06:14:11.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.222+0000 7f8235a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 0x7f82301590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.222+0000 7f8235a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8230073550 0x7f8230071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8235a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823006b630 con 0x7f8230073970 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8235a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f8230152d30 con 0x7f8230073550 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8235a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f823006b8d0 con 0x7f8230152950 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822f400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8230073970 0x7f823006cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55498/0 (socket says 172.21.15.132:55498) 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822f400640 1 -- 172.21.15.132:0/3951015973 learned_addr learned my addr 172.21.15.132:0/3951015973 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 0x7f82301590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8227e00640 1 --2- 172.21.15.132:0/3951015973 >> v2:172.21.15.134:3300/0 conn(0x7f8230073550 0x7f8230071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822e000640 1 -- 172.21.15.132:0/3951015973 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779873181 0 0) 0x7f823006b630 con 0x7f8230073970 2024-03-27T06:14:11.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822e000640 1 -- 172.21.15.132:0/3951015973 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8214003520 con 0x7f8230073970 2024-03-27T06:14:11.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8227e00640 1 -- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 msgr2=0x7f82301590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8227e00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 0x7f82301590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8227e00640 1 -- 172.21.15.132:0/3951015973 >> v1:172.21.15.134:6789/0 conn(0x7f8230073970 legacy=0x7f823006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f8227e00640 1 -- 172.21.15.132:0/3951015973 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823006bbb0 con 0x7f8230073550 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.223+0000 7f822ea00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 0x7f82301590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.224+0000 7f8227e00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230073550 0x7f8230071170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82180097a0 tx=0x7f821802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d5b1c71fbe5e4e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f822e000640 1 -- 172.21.15.132:0/3951015973 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218035020 con 0x7f8230073550 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f822e000640 1 -- 172.21.15.132:0/3951015973 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8218031810 con 0x7f8230073550 2024-03-27T06:14:11.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f822e000640 1 -- 172.21.15.132:0/3951015973 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218031be0 con 0x7f8230073550 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 -- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230073550 msgr2=0x7f8230071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230073550 0x7f8230071170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82180097a0 tx=0x7f821802b8b0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 -- 172.21.15.132:0/3951015973 shutdown_connections 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230073550 0x7f8230071170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 --2- 172.21.15.132:0/3951015973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230152950 0x7f82301590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 -- 172.21.15.132:0/3951015973 >> 172.21.15.132:0/3951015973 conn(0x7f82300cc060 msgr2=0x7f82300cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.225+0000 7f8235a00640 1 -- 172.21.15.132:0/3951015973 shutdown_connections 2024-03-27T06:14:11.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.226+0000 7f8235a00640 1 -- 172.21.15.132:0/3951015973 wait complete. 2024-03-27T06:14:11.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.227+0000 7f8235a00640 1 Processor -- start 2024-03-27T06:14:11.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.227+0000 7f8235a00640 1 -- start start 2024-03-27T06:14:11.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 0x7f82301f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 0x7f82301f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f823006c450 con 0x7f8230073970 2024-03-27T06:14:11.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f823006c170 con 0x7f8230152950 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.228+0000 7f8235a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f823006d660 con 0x7f8230073550 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 0x7f82301f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 0x7f82301f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:41180/0 (socket says 172.21.15.132:41180) 2024-03-27T06:14:11.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822ea00640 1 -- 172.21.15.132:0/4171074398 learned_addr learned my addr 172.21.15.132:0/4171074398 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 -- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 msgr2=0x7f823015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 -- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 msgr2=0x7f82301f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 0x7f82301f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 -- 172.21.15.132:0/4171074398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823014ead0 con 0x7f8230073970 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822ea00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:11.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f822f400640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 0x7f82301f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.229+0000 7f8227e00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 0x7f82301f36f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8230070b50 tx=0x7f82180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.230+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218035040 con 0x7f8230073970 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.230+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821803ad00 con 0x7f8230073970 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.230+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82180466a0 con 0x7f8230073970 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.230+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82301504c0 con 0x7f8230073970 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.230+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8230150a80 con 0x7f8230073970 2024-03-27T06:14:11.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.231+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8230068840 con 0x7f8230073970 2024-03-27T06:14:11.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.232+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8218038020 con 0x7f8230073970 2024-03-27T06:14:11.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.233+0000 7f8226a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 0x7f820c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.233+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f82180ad6e0 con 0x7f8230073970 2024-03-27T06:14:11.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.233+0000 7f822ea00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 0x7f820c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.234+0000 7f822ea00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 0x7f820c06ea20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82200095e0 tx=0x7f8220009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.238+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f821807b620 con 0x7f8230073970 2024-03-27T06:14:11.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.439+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82300c6b30 con 0x7f8230073970 2024-03-27T06:14:11.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.442+0000 7f8226a00640 1 -- 172.21.15.132:0/4171074398 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f821807f920 con 0x7f8230073970 2024-03-27T06:14:11.433 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:11.433 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:11.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 msgr2=0x7f820c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 0x7f820c06ea20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82200095e0 tx=0x7f8220009190 comp rx=0 tx=0).stop 2024-03-27T06:14:11.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 msgr2=0x7f82301f36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 0x7f82301f36f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8230070b50 tx=0x7f82180046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 shutdown_connections 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f820c06c560 0x7f820c06ea20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8230152950 0x7f82301f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8230073970 0x7f82301f36f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 --2- 172.21.15.132:0/4171074398 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8230073550 0x7f823015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 >> 172.21.15.132:0/4171074398 conn(0x7f82300cc060 msgr2=0x7f82300c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 shutdown_connections 2024-03-27T06:14:11.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.446+0000 7f8235a00640 1 -- 172.21.15.132:0/4171074398 wait complete. 2024-03-27T06:14:11.457 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T06:14:11.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:11.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.675+0000 7fc4dec00640 1 Processor -- start 2024-03-27T06:14:11.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.675+0000 7fc4dec00640 1 -- start start 2024-03-27T06:14:11.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.676+0000 7fc4dec00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc4d8152950 0x7fc4d81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.676+0000 7fc4dec00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d806b630 con 0x7fc4d8073970 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4dec00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d806b8d0 con 0x7fc4d8073550 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4dec00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d8152d30 con 0x7fc4d8152950 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4d7e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc4d8152950 0x7fc4d81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4cf400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc4d8073550 0x7fc4d8071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55502/0 (socket says 172.21.15.132:55502) 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4cf400640 1 -- 172.21.15.132:0/740512762 learned_addr learned my addr 172.21.15.132:0/740512762 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4d7400640 1 -- 172.21.15.132:0/740512762 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879930264 0 0) 0x7fc4d806b8d0 con 0x7fc4d8073550 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4d7e00640 1 -- 172.21.15.132:0/740512762 >> v1:172.21.15.134:6789/0 conn(0x7fc4d8073550 legacy=0x7fc4d8071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4d7e00640 1 -- 172.21.15.132:0/740512762 >> v1:172.21.15.132:6789/0 conn(0x7fc4d8073970 legacy=0x7fc4d806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.677+0000 7fc4d7e00640 1 -- 172.21.15.132:0/740512762 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d806bbb0 con 0x7fc4d8152950 2024-03-27T06:14:11.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.678+0000 7fc4d7e00640 1 --2- 172.21.15.132:0/740512762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8152950 0x7fc4d81590a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80097d0 tx=0x7fc4c802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b3d967ba41c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4d7400640 1 -- 172.21.15.132:0/740512762 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8037070 con 0x7fc4d8152950 2024-03-27T06:14:11.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4d7400640 1 -- 172.21.15.132:0/740512762 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8030710 con 0x7fc4d8152950 2024-03-27T06:14:11.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4d7400640 1 -- 172.21.15.132:0/740512762 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8030ae0 con 0x7fc4d8152950 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4dec00640 1 -- 172.21.15.132:0/740512762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8152950 msgr2=0x7fc4d81590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4dec00640 1 --2- 172.21.15.132:0/740512762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8152950 0x7fc4d81590a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80097d0 tx=0x7fc4c802b790 comp rx=0 tx=0).stop 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4dec00640 1 -- 172.21.15.132:0/740512762 shutdown_connections 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4dec00640 1 --2- 172.21.15.132:0/740512762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8152950 0x7fc4d81590a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.679+0000 7fc4dec00640 1 -- 172.21.15.132:0/740512762 >> 172.21.15.132:0/740512762 conn(0x7fc4d80cc060 msgr2=0x7fc4d80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:11.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.680+0000 7fc4dec00640 1 -- 172.21.15.132:0/740512762 shutdown_connections 2024-03-27T06:14:11.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.680+0000 7fc4dec00640 1 -- 172.21.15.132:0/740512762 wait complete. 2024-03-27T06:14:11.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.681+0000 7fc4dec00640 1 Processor -- start 2024-03-27T06:14:11.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.682+0000 7fc4dec00640 1 -- start start 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4d8073970 0x7fc4d81f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4d8152950 0x7fc4d806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d806c450 con 0x7fc4d8073550 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d806c170 con 0x7fc4d8073970 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d8071880 con 0x7fc4d8152950 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56830/0 (socket says 172.21.15.132:56830) 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 -- 172.21.15.132:0/3359718277 learned_addr learned my addr 172.21.15.132:0/3359718277 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4cf400640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4d8073970 0x7fc4d81f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4dcc00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4d8152950 0x7fc4d806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 -- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4d8152950 msgr2=0x7fc4d806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4d8152950 0x7fc4d806d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 -- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4d8073970 msgr2=0x7fc4d81f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4d8073970 0x7fc4d81f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.683+0000 7fc4d7e00640 1 -- 172.21.15.132:0/3359718277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d81f3dd0 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4d7e00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80097d0 tx=0x7fc4c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8037040 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8004030 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8004420 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d81f40b0 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.684+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d81f4670 con 0x7fc4d8073550 2024-03-27T06:14:11.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.685+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d8068840 con 0x7fc4d8073550 2024-03-27T06:14:11.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.686+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc4c8032940 con 0x7fc4d8073550 2024-03-27T06:14:11.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.687+0000 7fc4d4c00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 0x7fc4b406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:11.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.687+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc4c80ad450 con 0x7fc4d8073550 2024-03-27T06:14:11.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.688+0000 7fc4cf400640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 0x7fc4b406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:11.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.688+0000 7fc4cf400640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 0x7fc4b406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0007f70 tx=0x7fc4c0007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:11.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.692+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc4c8042020 con 0x7fc4d8073550 2024-03-27T06:14:11.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.896+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4d81e57e0 con 0x7fc4d8073550 2024-03-27T06:14:11.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.898+0000 7fc4d4c00640 1 -- 172.21.15.132:0/3359718277 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fc4c807b4b0 con 0x7fc4d8073550 2024-03-27T06:14:11.889 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:11.889 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 msgr2=0x7fc4b406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 0x7fc4b406e9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0007f70 tx=0x7fc4c0007f00 comp rx=0 tx=0).stop 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 msgr2=0x7fc4d814de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80097d0 tx=0x7fc4c80046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 shutdown_connections 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4d8152950 0x7fc4d806d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc4b406c530 0x7fc4b406e9f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4d8073970 0x7fc4d81f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 --2- 172.21.15.132:0/3359718277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4d8073550 0x7fc4d814de50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:11.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 >> 172.21.15.132:0/3359718277 conn(0x7fc4d80cc060 msgr2=0x7fc4d80c8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:11.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 shutdown_connections 2024-03-27T06:14:11.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:11.901+0000 7fc4dec00640 1 -- 172.21.15.132:0/3359718277 wait complete. 2024-03-27T06:14:11.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T06:14:11.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T06:14:11.913 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T06:14:11.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-27T06:14:11.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-27T06:14:11.914 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-27T06:14:11.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-27T06:14:11.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-27T06:14:12.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.222+0000 7f2e52200640 1 Processor -- start 2024-03-27T06:14:12.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.222+0000 7f2e52200640 1 -- start start 2024-03-27T06:14:12.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.223+0000 7f14ea200640 1 Processor -- start 2024-03-27T06:14:12.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.223+0000 7f14ea200640 1 -- start start 2024-03-27T06:14:12.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e44034390 0x7f2e4404b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2e440fff30 0x7f2e44100370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 0x7f2e440fc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2e4404b800 con 0x7f2e441008b0 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2e44006a00 con 0x7f2e440fff30 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.224+0000 7f2e52200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2e44006ce0 con 0x7f2e44034390 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2e440fff30 0x7f2e44100370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e50e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e44034390 0x7f2e4404b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2e440fff30 0x7f2e44100370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56152/0 (socket says 172.21.15.132:56152) 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 -- 172.21.15.132:0/3668977186 learned_addr learned my addr 172.21.15.132:0/3668977186 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 -- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e44034390 msgr2=0x7f2e4404b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 --2- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e44034390 0x7f2e4404b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.225+0000 7f2e4be00640 1 -- 172.21.15.132:0/3668977186 >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 msgr2=0x7f2e440fc940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f359c000640 1 Processor -- start 2024-03-27T06:14:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f14ea200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f14dc00e260 0x7f14dc04b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f359c000640 1 -- start start 2024-03-27T06:14:12.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f14ea200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f14dc0d51c0 0x7f14dc0452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f2e51800640 1 --2- 172.21.15.132:0/3668977186 >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 0x7f2e440fc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7fc8f9c00640 1 Processor -- start 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f14ea200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14dc045e90 con 0x7f14dc04b050 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f14ea200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f14dc046ef0 con 0x7f14dc00e260 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.226+0000 7f14ea200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f14dc046130 con 0x7f14dc0d51c0 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f14dc0d51c0 0x7f14dc0452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f14dc0d51c0 0x7f14dc0452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56166/0 (socket says 172.21.15.132:56166) 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 -- 172.21.15.132:0/1191656358 learned_addr learned my addr 172.21.15.132:0/1191656358 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f2e4be00640 1 --2- 172.21.15.132:0/3668977186 >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 0x7f2e440fc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f2e4be00640 1 -- 172.21.15.132:0/3668977186 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e44006fc0 con 0x7f2e440fff30 2024-03-27T06:14:12.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f2e51800640 1 --2- 172.21.15.132:0/3668977186 >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 0x7f2e440fc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 -- 172.21.15.132:0/1191656358 >> v2:172.21.15.132:3300/0 conn(0x7f14dc00e260 msgr2=0x7f14dc04b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 --2- 172.21.15.132:0/1191656358 >> v2:172.21.15.132:3300/0 conn(0x7f14dc00e260 0x7f14dc04b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7f14e9800640 1 -- 172.21.15.132:0/1191656358 >> v1:172.21.15.132:6789/0 conn(0x7f14dc04b050 legacy=0x7f14dc0db7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.227+0000 7fc8f9c00640 1 -- start start 2024-03-27T06:14:12.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.228+0000 7f359c000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f358c0d51f0 0x7f358c04b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.228+0000 7f2e4be00640 1 --2- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e440fff30 0x7f2e44100370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e440084f0 tx=0x7f2e3802db20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589d9021be4e7153 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.228+0000 7f14e9800640 1 -- 172.21.15.132:0/1191656358 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14dc046410 con 0x7f14dc0d51c0 2024-03-27T06:14:12.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f2e4b400640 1 -- 172.21.15.132:0/3668977186 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3802f8e0 con 0x7f2e440fff30 2024-03-27T06:14:12.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f359c000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c045320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f2e4b400640 1 -- 172.21.15.132:0/3668977186 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3802fa80 con 0x7f2e440fff30 2024-03-27T06:14:12.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f2e4b400640 1 -- 172.21.15.132:0/3668977186 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3802fe50 con 0x7f2e440fff30 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f359c000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358c045ec0 con 0x7f358c04b080 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f359c000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f358c046f20 con 0x7f358c0d51f0 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f3598c00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f358c04b080 0x7f358c0db820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:32972/0 (socket says 172.21.15.132:32972) 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f359c000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f358c046160 con 0x7f358c005e80 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f3598c00640 1 -- 172.21.15.132:0/1949485358 learned_addr learned my addr 172.21.15.132:0/1949485358 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f3599600640 1 --2- 172.21.15.132:0/1949485358 >> v2:172.21.15.134:3300/0 conn(0x7f358c0d51f0 0x7f358c04b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.229+0000 7f359a000640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c045320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 -- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e440fff30 msgr2=0x7f2e44100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 --2- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e440fff30 0x7f2e44100370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e440084f0 tx=0x7f2e3802db20 comp rx=0 tx=0).stop 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f9c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc8ec0e0480 0x7fc8ec039e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f9c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ec03a4a0 con 0x7fc8ec0d9240 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f9c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ec04ef20 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f9c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fc8ec04f200 con 0x7fc8ec0e0480 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 -- 172.21.15.132:0/3668977186 shutdown_connections 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 --2- 172.21.15.132:0/3668977186 >> v2:172.21.15.132:3300/0 conn(0x7f2e441008b0 0x7f2e440fc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 --2- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e440fff30 0x7f2e44100370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 --2- 172.21.15.132:0/3668977186 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e44034390 0x7f2e4404b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f2e52200640 1 -- 172.21.15.132:0/3668977186 >> 172.21.15.132:0/3668977186 conn(0x7f2e44050140 msgr2=0x7f2e44050550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f2a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fc8ec0d9240 0x7fc8ec0071a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:32982/0 (socket says 172.21.15.132:32982) 2024-03-27T06:14:12.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7fc8f2a00640 1 -- 172.21.15.132:0/3618454092 learned_addr learned my addr 172.21.15.132:0/3618454092 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f2e52200640 1 -- 172.21.15.132:0/3668977186 shutdown_connections 2024-03-27T06:14:12.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.230+0000 7f3599600640 1 -- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 msgr2=0x7f358c045320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f14e9800640 1 --2- 172.21.15.132:0/1191656358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc0d51c0 0x7f14dc0452f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14e406b250 tx=0x7f14e40abfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6b7269f09b74f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f2e52200640 1 -- 172.21.15.132:0/3668977186 wait complete. 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7fc8f2000640 1 --2- 172.21.15.132:0/3618454092 >> v2:172.21.15.132:3300/0 conn(0x7fc8ec0e0480 0x7fc8ec039e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351635556 0 0) 0x7fc8ec04ef20 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d8003520 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f3599600640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c045320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f3599600640 1 -- 172.21.15.132:0/1949485358 >> v1:172.21.15.132:6789/0 conn(0x7f358c04b080 legacy=0x7f358c0db820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f3599600640 1 -- 172.21.15.132:0/1949485358 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358c046440 con 0x7f358c0d51f0 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.231+0000 7f359a000640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c045320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 415278097 0 0) 0x7fc8d8003520 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 >> v2:172.21.15.132:3300/0 conn(0x7fc8ec0e0480 msgr2=0x7fc8ec039e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 --2- 172.21.15.132:0/3618454092 >> v2:172.21.15.132:3300/0 conn(0x7fc8ec0e0480 0x7fc8ec039e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 >> v1:172.21.15.132:6789/0 conn(0x7fc8ec0d9240 legacy=0x7fc8ec0071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec04f4e0 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.233+0000 7f2e52200640 1 Processor -- start 2024-03-27T06:14:12.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7f14e3400640 1 -- 172.21.15.132:0/1191656358 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e40b2070 con 0x7f14dc0d51c0 2024-03-27T06:14:12.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7f14e3400640 1 -- 172.21.15.132:0/1191656358 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e40b6400 con 0x7f14dc0d51c0 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4217832196 0 0) 0x7fc8f40686c0 con 0x7fc8ec04ffc0 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 >> v1:172.21.15.134:6789/0 conn(0x7fc8ec04ffc0 legacy=0x7fc8ec0e98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 0x7fc8d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80043c0 0x7fc8d80048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8d80050a0 0x7fc8d80141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec04f200 con 0x7fc8d80050a0 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d8014770 con 0x7fc8d80039b0 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d8014970 con 0x7fc8d80043c0 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.232+0000 7fc8f2000640 1 --2- 172.21.15.132:0/3618454092 >> v2:172.21.15.132:3300/0 conn(0x7fc8ec0e0480 0x7fc8ec039e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:12.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.233+0000 7f3599600640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c04b610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f358800a840 tx=0x7f358800a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7ee1fdf8528658e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.233+0000 7f2e52200640 1 -- start start 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 0x7f2e440e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e440fff30 0x7f2e44196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e441008b0 0x7f2e441970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2e44007580 con 0x7f2e441008b0 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e44100d20 con 0x7f2e44034390 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f2e52200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4404c6c0 con 0x7f2e440fff30 2024-03-27T06:14:12.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 0x7fc8d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f14e3400640 1 -- 172.21.15.132:0/1191656358 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e40b6870 con 0x7f14dc0d51c0 2024-03-27T06:14:12.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f3593e00640 1 -- 172.21.15.132:0/1949485358 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588032070 con 0x7f358c0d51f0 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f14ea200640 1 -- 172.21.15.132:0/1191656358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc0d51c0 msgr2=0x7f14dc0452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7f14ea200640 1 --2- 172.21.15.132:0/1191656358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc0d51c0 0x7f14dc0452f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14e406b250 tx=0x7f14e40abfe0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7fc8f2000640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80043c0 0x7fc8d80048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7fc8f2a00640 1 -- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80043c0 msgr2=0x7fc8d80048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80043c0 0x7fc8d80048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.234+0000 7fc8f2a00640 1 -- 172.21.15.132:0/3618454092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8d80050a0 msgr2=0x7fc8d80141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7fc8f3400640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8d80050a0 0x7fc8d80141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f2e50e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 0x7f2e440e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f2e50e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 0x7f2e440e8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56198/0 (socket says 172.21.15.132:56198) 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f2e50e00640 1 -- 172.21.15.132:0/949397829 learned_addr learned my addr 172.21.15.132:0/949397829 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 -- 172.21.15.132:0/1191656358 shutdown_connections 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 --2- 172.21.15.132:0/1191656358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc0d51c0 0x7f14dc0452f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 --2- 172.21.15.132:0/1191656358 >> v2:172.21.15.132:3300/0 conn(0x7f14dc00e260 0x7f14dc04b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 -- 172.21.15.132:0/1191656358 >> 172.21.15.132:0/1191656358 conn(0x7f14dc050140 msgr2=0x7f14dc050550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f2e4be00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e440fff30 0x7f2e44196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 -- 172.21.15.132:0/1191656358 shutdown_connections 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8d80050a0 0x7fc8d80141d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7fc8f2a00640 1 -- 172.21.15.132:0/3618454092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d8003520 con 0x7fc8d80039b0 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f3593e00640 1 -- 172.21.15.132:0/1949485358 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f358802cb00 con 0x7f358c0d51f0 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f3593e00640 1 -- 172.21.15.132:0/1949485358 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588036400 con 0x7f358c0d51f0 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f14ea200640 1 -- 172.21.15.132:0/1191656358 wait complete. 2024-03-27T06:14:12.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7f2e51800640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e441008b0 0x7f2e441970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.235+0000 7fc8f3400640 1 -- 172.21.15.132:0/3618454092 reap_dead start 2024-03-27T06:14:12.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 -- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e440fff30 msgr2=0x7f2e44196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2e440fff30 0x7f2e44196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 -- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e441008b0 msgr2=0x7f2e441970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e441008b0 0x7f2e441970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e44049fd0 con 0x7f2e44034390 2024-03-27T06:14:12.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.236+0000 7f359c000640 1 -- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 msgr2=0x7f358c04b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.236+0000 7f359c000640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c04b610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f358800a840 tx=0x7f358800a1a0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e50e00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 0x7f2e440e8af0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d830 tx=0x7f2e4000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40018020 con 0x7f2e44034390 2024-03-27T06:14:12.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e400096a0 con 0x7f2e44034390 2024-03-27T06:14:12.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40009a70 con 0x7f2e44034390 2024-03-27T06:14:12.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.237+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 0x7fc8d8003d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4001970 tx=0x7fc8d8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d93290072854e7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e44199dd0 con 0x7f2e44034390 2024-03-27T06:14:12.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4037070 con 0x7fc8d80039b0 2024-03-27T06:14:12.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7fd2f3200640 1 Processor -- start 2024-03-27T06:14:12.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e4005ad0 con 0x7fc8d80039b0 2024-03-27T06:14:12.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7fd2f3200640 1 -- start start 2024-03-27T06:14:12.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7fc8f1600640 1 -- 172.21.15.132:0/3618454092 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4001540 con 0x7fc8d80039b0 2024-03-27T06:14:12.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4419a370 con 0x7f2e44034390 2024-03-27T06:14:12.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f359c000640 1 -- 172.21.15.132:0/1949485358 shutdown_connections 2024-03-27T06:14:12.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f359c000640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c045320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f14ea200640 1 Processor -- start 2024-03-27T06:14:12.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f359c000640 1 --2- 172.21.15.132:0/1949485358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c04b610 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.238+0000 7f359c000640 1 -- 172.21.15.132:0/1949485358 >> 172.21.15.132:0/1949485358 conn(0x7f358c050170 msgr2=0x7f358c050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7fa35a400640 1 Processor -- start 2024-03-27T06:14:12.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7f5581c00640 1 Processor -- start 2024-03-27T06:14:12.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7f5581c00640 1 -- start start 2024-03-27T06:14:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e28001030 con 0x7f2e44034390 2024-03-27T06:14:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7fa35a400640 1 -- start start 2024-03-27T06:14:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7fc8f9c00640 1 -- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 msgr2=0x7fc8d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7fc8f9c00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 0x7fc8d8003d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4001970 tx=0x7fc8d8005830 comp rx=0 tx=0).stop 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.239+0000 7f14ea200640 1 -- start start 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fc8f9c00640 1 -- 172.21.15.132:0/3618454092 shutdown_connections 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fc8f9c00640 1 --2- 172.21.15.132:0/3618454092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8d80039b0 0x7fc8d8003d90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fc8f9c00640 1 -- 172.21.15.132:0/3618454092 >> 172.21.15.132:0/3618454092 conn(0x7fc8ec00e150 msgr2=0x7fc8ec00e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec06c8a0 0x7fd2ec0c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7f359c000640 1 -- 172.21.15.132:0/1949485358 shutdown_connections 2024-03-27T06:14:12.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fc8f9c00640 1 -- 172.21.15.132:0/3618454092 shutdown_connections 2024-03-27T06:14:12.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fc8f9c00640 1 -- 172.21.15.132:0/3618454092 wait complete. 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f3200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd2ec06cc80 0x7fd2ec0c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f3200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec0cdad0 con 0x7fd2ec0c77f0 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f3200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd2ec0cc210 con 0x7fd2ec06cc80 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f3200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec15b690 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2eb400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd2ec06cc80 0x7fd2ec0c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f1200640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd2ec0c77f0 0x7fd2ec154890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:32988/0 (socket says 172.21.15.132:32988) 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.240+0000 7fd2f1200640 1 -- 172.21.15.132:0/385782173 learned_addr learned my addr 172.21.15.132:0/385782173 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fd2eb400640 1 -- 172.21.15.132:0/385782173 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec06c8a0 msgr2=0x7fd2ec0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fa35a400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 0x7fa35415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fa35a400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fa35a400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3540cb9f0 con 0x7fa3540d0970 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fa35a400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa35415e2e0 con 0x7fa3540d0d50 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7fa35a400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa35415e5c0 con 0x7fa3540cb440 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.241+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2e4001b020 con 0x7f2e44034390 2024-03-27T06:14:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f359c000640 1 -- 172.21.15.132:0/1949485358 wait complete. 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fa353400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa3540d0970 0x7fa354154aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37712/0 (socket says 172.21.15.132:37712) 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fa353400640 1 -- 172.21.15.132:0/116861386 learned_addr learned my addr 172.21.15.132:0/116861386 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fa353e00640 1 --2- 172.21.15.132:0/116861386 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fd2ebe00640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec06c8a0 0x7fd2ec0c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fd2eb400640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec06c8a0 0x7fd2ec0c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fd2eb400640 1 -- 172.21.15.132:0/385782173 >> v1:172.21.15.132:6789/0 conn(0x7fd2ec0c77f0 legacy=0x7fd2ec154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fd2eb400640 1 -- 172.21.15.132:0/385782173 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec15b970 con 0x7fd2ec06cc80 2024-03-27T06:14:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f5581c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7fa353e00640 1 -- 172.21.15.132:0/116861386 >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 msgr2=0x7fa35415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f5581c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 0x7f557c0c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f5581c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c0cdad0 con 0x7f557c0c77f0 2024-03-27T06:14:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f5581c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f557c0cc210 con 0x7f557c06cc80 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f5581c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f557c15b690 con 0x7f557c06c8a0 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f557aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f557aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35122/0 (socket says 172.21.15.132:35122) 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.242+0000 7f557aa00640 1 -- 172.21.15.132:0/1213075615 learned_addr learned my addr 172.21.15.132:0/1213075615 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f2e48c00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2e2c06c530 0x7f2e2c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2e4008b760 con 0x7f2e44034390 2024-03-27T06:14:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7fa352a00640 1 --2- 172.21.15.132:0/116861386 >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 0x7fa35415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f557a000640 1 --2- 172.21.15.132:0/1213075615 >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 0x7f557c0c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f2e48c00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7f2e2c074730 0x7f2e2c076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e40010e50 con 0x7f2e2c074730 2024-03-27T06:14:12.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f557aa00640 1 -- 172.21.15.132:0/1213075615 >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 msgr2=0x7f557c0c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f557aa00640 1 --2- 172.21.15.132:0/1213075615 >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 0x7f557c0c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f2e4be00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2e2c06c530 0x7f2e2c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7f557aa00640 1 -- 172.21.15.132:0/1213075615 >> v1:172.21.15.134:6789/0 conn(0x7f557c0c77f0 legacy=0x7f557c154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f2e51800640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7f2e2c074730 0x7f2e2c076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7fc8f9c00640 1 Processor -- start 2024-03-27T06:14:12.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e40092050 con 0x7f2e44034390 2024-03-27T06:14:12.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7fd2eb400640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 0x7fd2ec0c8860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95f9c9c62cb1c42 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7fa353e00640 1 --2- 172.21.15.132:0/116861386 >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 0x7fa35415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7fa353e00640 1 -- 172.21.15.132:0/116861386 >> v1:172.21.15.134:6789/0 conn(0x7fa3540d0970 legacy=0x7fa354154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.243+0000 7fa353e00640 1 -- 172.21.15.132:0/116861386 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35415e8a0 con 0x7fa3540cb440 2024-03-27T06:14:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f2e51800640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7f2e2c074730 0x7f2e2c076b50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f14ea200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 0x7f14dc1816f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14ea200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc04b050 0x7f14dc0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14ea200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14dc0d51c0 0x7f14dc0dea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14e8e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 0x7f14dc1816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14e8e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 0x7f14dc1816f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35138/0 (socket says 172.21.15.132:35138) 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14e8e00640 1 -- 172.21.15.132:0/1486810240 learned_addr learned my addr 172.21.15.132:0/1486810240 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc045830 con 0x7f14dc0d51c0 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc0469d0 con 0x7f14dc04b050 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f14dc0459b0 con 0x7f14dc00e260 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7f359c000640 1 Processor -- start 2024-03-27T06:14:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f5579600640 1 -- 172.21.15.132:0/1213075615 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683894476 0 0) 0x7f557c0cdad0 con 0x7f557c0c77f0 2024-03-27T06:14:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f557aa00640 1 -- 172.21.15.132:0/1213075615 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c15b970 con 0x7f557c06c8a0 2024-03-27T06:14:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7f557a000640 1 --2- 172.21.15.132:0/1213075615 >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 0x7f557c0c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.245+0000 7fa352a00640 1 --2- 172.21.15.132:0/116861386 >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 0x7fa35415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== osd.0 v2:172.21.15.132:6800/1970178383 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e40010e50 con 0x7f2e2c074730 2024-03-27T06:14:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fd2eaa00640 1 -- 172.21.15.132:0/385782173 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8031780 con 0x7fd2ec06cc80 2024-03-27T06:14:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fd2eaa00640 1 -- 172.21.15.132:0/385782173 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d8037070 con 0x7fd2ec06cc80 2024-03-27T06:14:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fd2eaa00640 1 -- 172.21.15.132:0/385782173 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8031b40 con 0x7fd2ec06cc80 2024-03-27T06:14:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fa353e00640 1 --2- 172.21.15.132:0/116861386 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540c8a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009d90 tx=0x7fa34c02bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c4205ffb454570 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f557aa00640 1 --2- 172.21.15.132:0/1213075615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f556c009870 tx=0x7f556c02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=878362b0677d2f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e3e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc04b050 0x7f14dc0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 -- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc04b050 msgr2=0x7f14dc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fa352000640 1 -- 172.21.15.132:0/116861386 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c039070 con 0x7fa3540cb440 2024-03-27T06:14:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc04b050 0x7f14dc0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 -- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14dc0d51c0 msgr2=0x7f14dc0dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14dc0d51c0 0x7f14dc0dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14dc049210 con 0x7f14dc00e260 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e3e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14dc04b050 0x7f14dc0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f14e8e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 0x7f14dc1816f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b410 tx=0x7f14d800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7f359c000640 1 -- start start 2024-03-27T06:14:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f5579600640 1 -- 172.21.15.132:0/1213075615 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c035020 con 0x7f557c06c8a0 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f5579600640 1 -- 172.21.15.132:0/1213075615 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c031870 con 0x7f557c06c8a0 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f5579600640 1 -- 172.21.15.132:0/1213075615 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c031c80 con 0x7f557c06c8a0 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f2e4be00640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2e2c06c530 0x7f2e2c06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e38038000 tx=0x7f2e38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8015020 con 0x7f14dc00e260 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d8011830 con 0x7f14dc00e260 2024-03-27T06:14:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8011c20 con 0x7f14dc00e260 2024-03-27T06:14:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fa352000640 1 -- 172.21.15.132:0/116861386 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c02cc70 con 0x7fa3540cb440 2024-03-27T06:14:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fa352000640 1 -- 172.21.15.132:0/116861386 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c02ce60 con 0x7fa3540cb440 2024-03-27T06:14:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14dc0494b0 con 0x7f14dc00e260 2024-03-27T06:14:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14dc049a50 con 0x7f14dc00e260 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.244+0000 7fc8f9c00640 1 -- start start 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 0x7fc8ec006f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8ec170ef0 0x7fc8ec049690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ec048770 0x7fc8ec049bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec050410 con 0x7fc8ec048770 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec04faa0 con 0x7fc8ec170ef0 2024-03-27T06:14:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.246+0000 7fc8f9c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc8ec0d9620 con 0x7fc8d80039b0 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 0x7fc8ec006f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 0x7fc8ec006f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35148/0 (socket says 172.21.15.132:35148) 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 -- 172.21.15.132:0/1947215825 learned_addr learned my addr 172.21.15.132:0/1947215825 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 -- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8ec170ef0 msgr2=0x7fc8ec049690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8ec170ef0 0x7fc8ec049690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 -- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ec048770 msgr2=0x7fc8ec049bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ec048770 0x7fc8ec049bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.247+0000 7fc8f2a00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec04af10 con 0x7fc8d80039b0 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fc8f2a00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 0x7fc8ec006f10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4001970 tx=0x7fc8e4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4037040 con 0x7fc8d80039b0 2024-03-27T06:14:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ec0d9b20 con 0x7fc8d80039b0 2024-03-27T06:14:12.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 -- 172.21.15.132:0/1213075615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 msgr2=0x7f557c0c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e4032470 con 0x7fc8d80039b0 2024-03-27T06:14:12.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 --2- 172.21.15.132:0/1213075615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f556c009870 tx=0x7f556c02b910 comp rx=0 tx=0).stop 2024-03-27T06:14:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fd2f3200640 1 -- 172.21.15.132:0/385782173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 msgr2=0x7fd2ec0c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 -- 172.21.15.132:0/1213075615 shutdown_connections 2024-03-27T06:14:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7fd2f3200640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 0x7fd2ec0c8860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b820 comp rx=0 tx=0).stop 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 --2- 172.21.15.132:0/1213075615 >> v2:172.21.15.132:3300/0 conn(0x7f557c06cc80 0x7f557c0c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 --2- 172.21.15.132:0/1213075615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06c8a0 0x7f557c0c8320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.248+0000 7f5581c00640 1 -- 172.21.15.132:0/1213075615 >> 172.21.15.132:0/1213075615 conn(0x7f557c0ccbe0 msgr2=0x7f557c0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c049a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fd2f3200640 1 -- 172.21.15.132:0/385782173 shutdown_connections 2024-03-27T06:14:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 0x7f358c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f5581c00640 1 -- 172.21.15.132:0/1213075615 shutdown_connections 2024-03-27T06:14:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ec0da0e0 con 0x7fc8d80039b0 2024-03-27T06:14:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fd2f3200640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 0x7fd2ec0c8860 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fd2f3200640 1 --2- 172.21.15.132:0/385782173 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec06c8a0 0x7fd2ec0c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fd2f3200640 1 -- 172.21.15.132:0/385782173 >> 172.21.15.132:0/385782173 conn(0x7fd2ec0ccbe0 msgr2=0x7fd2ec0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fa35a400640 1 -- 172.21.15.132:0/116861386 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 msgr2=0x7fa3540c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f358c00e290 con 0x7f358c04b080 2024-03-27T06:14:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f358c046a00 con 0x7f358c0d51f0 2024-03-27T06:14:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f359c000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f358c00e410 con 0x7f358c005e80 2024-03-27T06:14:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f5581c00640 1 -- 172.21.15.132:0/1213075615 wait complete. 2024-03-27T06:14:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7fa35a400640 1 --2- 172.21.15.132:0/116861386 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540c8a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009d90 tx=0x7fa34c02bec0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14dc005350 con 0x7f14dc00e260 2024-03-27T06:14:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4037040 con 0x7fc8d80039b0 2024-03-27T06:14:12.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fd2f3200640 1 -- 172.21.15.132:0/385782173 shutdown_connections 2024-03-27T06:14:12.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 -- 172.21.15.132:0/116861386 shutdown_connections 2024-03-27T06:14:12.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc8ec0057d0 con 0x7fc8d80039b0 2024-03-27T06:14:12.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 --2- 172.21.15.132:0/116861386 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540c8a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f3598c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 0x7f358c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 --2- 172.21.15.132:0/116861386 >> v2:172.21.15.134:3300/0 conn(0x7fa3540d0d50 0x7fa35415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f3598c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 0x7f358c0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37354/0 (socket says 172.21.15.132:37354) 2024-03-27T06:14:12.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f3598c00640 1 -- 172.21.15.132:0/2358779695 learned_addr learned my addr 172.21.15.132:0/2358779695 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f359a000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 -- 172.21.15.132:0/116861386 >> 172.21.15.132:0/116861386 conn(0x7fa35406dbd0 msgr2=0x7fa35406dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fd2f3200640 1 -- 172.21.15.132:0/385782173 wait complete. 2024-03-27T06:14:12.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.249+0000 7f3599600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c049a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 -- 172.21.15.132:0/116861386 shutdown_connections 2024-03-27T06:14:12.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7fa35a400640 1 -- 172.21.15.132:0/116861386 wait complete. 2024-03-27T06:14:12.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f3e9d800640 1 Processor -- start 2024-03-27T06:14:12.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f3e9d800640 1 -- start start 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f359a000640 1 -- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 msgr2=0x7f358c049a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f359a000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c049a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.250+0000 7f359a000640 1 -- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 msgr2=0x7f358c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f359a000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 0x7f358c0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f359a000640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358c0cc290 con 0x7f358c0d51f0 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f359a000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c0cd130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3594050b20 tx=0x7f3594069930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f3599600640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f358c005e80 0x7f358c049a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594091070 con 0x7f358c0d51f0 2024-03-27T06:14:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f359406b9f0 con 0x7f358c0d51f0 2024-03-27T06:14:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.251+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359406bda0 con 0x7f358c0d51f0 2024-03-27T06:14:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f358c0cbfb0 con 0x7f358c0d51f0 2024-03-27T06:14:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e980c8a30 0x7f3e98154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e9d800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e980cdad0 con 0x7f3e9806c8a0 2024-03-27T06:14:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e9d800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e9815b690 con 0x7f3e9806cc80 2024-03-27T06:14:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3598c00640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358c04b080 0x7f358c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc8e4032620 con 0x7fc8d80039b0 2024-03-27T06:14:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e9d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3e980cc210 con 0x7f3e980c8a30 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e96a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f3e9806c8a0 0x7f3e980c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:32994/0 (socket says 172.21.15.132:32994) 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f3e96a00640 1 -- 172.21.15.132:0/3307394720 learned_addr learned my addr 172.21.15.132:0/3307394720 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310680391 0 0) 0x7f3e980cdad0 con 0x7f3e9806c8a0 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e7c003520 con 0x7f3e9806c8a0 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.252+0000 7f5581c00640 1 Processor -- start 2024-03-27T06:14:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f5581c00640 1 -- start start 2024-03-27T06:14:12.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f14d8018020 con 0x7f14dc00e260 2024-03-27T06:14:12.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7fd2f3200640 1 Processor -- start 2024-03-27T06:14:12.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7fa35a400640 1 Processor -- start 2024-03-27T06:14:12.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7fa35a400640 1 -- start start 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f358c048380 con 0x7f358c0d51f0 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f14e0c00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f14c406c600 0x7f14c406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f14d808b630 con 0x7f14dc00e260 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f14e0c00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7f14c4074880 0x7f14c4076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14d8008eb0 con 0x7f14c4074880 2024-03-27T06:14:12.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f14d8092050 con 0x7f14dc00e260 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7fc8eaa00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc8d406c760 0x7fc8d406ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc8e40ad7e0 con 0x7fc8d80039b0 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7fc8eaa00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fc8d4074960 0x7fc8d4076d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8e4038e50 con 0x7fc8d4074960 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc8e40adbd0 con 0x7fc8d80039b0 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7fc8f2000640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc8d406c760 0x7fc8d406ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7fd2f3200640 1 -- start start 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7fc8f3400640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fc8d4074960 0x7fc8d4076d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7fc8f2000640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc8d406c760 0x7fc8d406ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec04b330 tx=0x7fc8e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 708061621 0 0) 0x7f3e7c003520 con 0x7f3e9806c8a0 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e980c8a30 msgr2=0x7f3e98154fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e980c8a30 0x7f3e98154fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 >> v1:172.21.15.134:6789/0 conn(0x7f3e9806cc80 legacy=0x7f3e98154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9815b970 con 0x7f3e9806c8a0 2024-03-27T06:14:12.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4217832196 0 0) 0x7f3e88001670 con 0x7f3e9806c8a0 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 >> v1:172.21.15.132:6789/0 conn(0x7f3e9806c8a0 legacy=0x7f3e980c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e7c0039e0 0x7f3e7c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e7c004800 0x7f3e7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0052e0 0x7f3e7c0141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3e980cc210 con 0x7f3e7c004800 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c014a10 con 0x7f3e7c0052e0 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.253+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c014c10 con 0x7f3e7c0039e0 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f3e96a00640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e7c0039e0 0x7f3e7c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f3e96a00640 1 -- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0052e0 msgr2=0x7f3e7c0141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f3e96a00640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0052e0 0x7f3e7c0141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f3e96a00640 1 -- 172.21.15.132:0/3307394720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e7c004800 msgr2=0x7f3e7c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f3e96a00640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e7c004800 0x7f3e7c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.255+0000 7f3e96a00640 1 -- 172.21.15.132:0/3307394720 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9815b970 con 0x7f3e7c0039e0 2024-03-27T06:14:12.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f359409c060 con 0x7f358c0d51f0 2024-03-27T06:14:12.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 0x7f557c0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f358c005380 con 0x7f358c0d51f0 2024-03-27T06:14:12.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06cc80 0x7f557c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f14e9800640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7f14c4074880 0x7f14c4076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c0c77f0 0x7f557c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f557c06d440 con 0x7f557c06c8a0 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f557c06d2c0 con 0x7f557c0c77f0 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.254+0000 7f5581c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f557c0cd700 con 0x7f557c06cc80 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7f557aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 0x7f557c0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7f557aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 0x7f557c0c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37360/0 (socket says 172.21.15.132:37360) 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.256+0000 7f557aa00640 1 -- 172.21.15.132:0/302766146 learned_addr learned my addr 172.21.15.132:0/302766146 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 -- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06cc80 msgr2=0x7f557c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06cc80 0x7f557c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 -- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c0c77f0 msgr2=0x7f557c15c7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c0c77f0 0x7f557c15c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c1f4260 con 0x7f557c06c8a0 2024-03-27T06:14:12.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7fa35a400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7fa35a400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3540d0970 0x7fa3540cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7fa35a400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 0x7fa3540cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f3591600640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f357406c580 0x7f357406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 0x7fd2ec1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 0x7fd2ec15e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec0c77f0 0x7fd2ec15bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec06d440 con 0x7fd2ec06cc80 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec06d2c0 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2f3200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec0cd700 con 0x7fd2ec0c77f0 2024-03-27T06:14:12.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7f14e3e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f14c406c600 0x7f14c406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f557aa00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 0x7f557c0c82e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f556c009990 tx=0x7f556c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 0x7fd2ec1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 0x7fd2ec1f1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56264/0 (socket says 172.21.15.132:56264) 2024-03-27T06:14:12.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fd2ebe00640 1 -- 172.21.15.132:0/2629381480 learned_addr learned my addr 172.21.15.132:0/2629381480 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7f3e96a00640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e7c0039e0 0x7f3e7c003e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e8801d290 tx=0x7f3e7c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e1749981d65b6b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fa353400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fc8f3400640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fc8d4074960 0x7fc8d4076d80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c040070 con 0x7f557c06c8a0 2024-03-27T06:14:12.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c03a420 con 0x7f557c06c8a0 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c035040 con 0x7f557c06c8a0 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f5581c00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c1f44e0 con 0x7f557c06c8a0 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fa353400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35186/0 (socket says 172.21.15.132:35186) 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fa352a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3540d0970 0x7fa3540cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fa353e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 0x7fa3540cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7fa35a400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa354155330 con 0x7fa3540d0d50 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fa353e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 0x7fa3540cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37366/0 (socket says 172.21.15.132:37366) 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fa353e00640 1 -- 172.21.15.132:0/1254595944 learned_addr learned my addr 172.21.15.132:0/1254595944 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa3541551b0 con 0x7fa3540d0970 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa3540cc7f0 con 0x7fa3540cb440 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fd2ebe00640 1 -- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec0c77f0 msgr2=0x7fd2ec15bcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fd2ebe00640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd2ec0c77f0 0x7fd2ec15bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fd2ebe00640 1 -- 172.21.15.132:0/2629381480 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 msgr2=0x7fd2ec15e240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fd2ebe00640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2ec06cc80 0x7fd2ec15e240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7fd2ebe00640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec15ca60 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f14e9800640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7f14c4074880 0x7f14c4076ca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f14e3e00640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f14c406c600 0x7f14c406eac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0d2a90 tx=0x7f14d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f35941067d0 con 0x7f358c0d51f0 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f3591600640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7f3574074780 0x7f3574076ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.257+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3594071e50 con 0x7f3574074780 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f359410d050 con 0x7f358c0d51f0 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.258+0000 7f3599600640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f357406c580 0x7f357406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f3598c00640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7f3574074780 0x7f3574076ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8804a070 con 0x7f3e7c0039e0 2024-03-27T06:14:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.259+0000 7f5581c00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c1f4aa0 con 0x7f557c06c8a0 2024-03-27T06:14:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7f5581c00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f557c06d080 con 0x7f557c06c8a0 2024-03-27T06:14:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== osd.1 v2:172.21.15.132:6813/3659088073 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14d8008eb0 con 0x7f14c4074880 2024-03-27T06:14:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8803dbb0 con 0x7f3e7c0039e0 2024-03-27T06:14:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7f3e95600640 1 -- 172.21.15.132:0/3307394720 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88018d60 con 0x7f3e7c0039e0 2024-03-27T06:14:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fd2ebe00640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 0x7fd2ec1f1370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00ce10 tx=0x7fd2dc00a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fa353e00640 1 -- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 msgr2=0x7fa3540b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fa353e00640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa3540cb440 0x7fa3540b3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fa353e00640 1 -- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3540d0970 msgr2=0x7fa3540cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fa353e00640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3540d0970 0x7fa3540cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fa353e00640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3541481f0 con 0x7fa3540d0d50 2024-03-27T06:14:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc003b90 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2dc003d30 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc004240 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== osd.3 v2:172.21.15.132:6808/2165878941 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8e4038e50 con 0x7fc8d4074960 2024-03-27T06:14:12.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.260+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec15c780 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.261+0000 7fa353e00640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 0x7fa3540cfc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009d90 tx=0x7fa34c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.261+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ec152570 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7f3e9d800640 1 -- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e7c0039e0 msgr2=0x7f3e7c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7f3e9d800640 1 --2- 172.21.15.132:0/3307394720 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e7c0039e0 0x7f3e7c003e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e8801d290 tx=0x7f3e7c005830 comp rx=0 tx=0).stop 2024-03-27T06:14:12.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7f3e97400640 1 -- 172.21.15.132:0/3307394720 reap_dead start 2024-03-27T06:14:12.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7f3e9d800640 1 -- 172.21.15.132:0/3307394720 shutdown_connections 2024-03-27T06:14:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.261+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c039050 con 0x7fa3540d0d50 2024-03-27T06:14:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa354154070 con 0x7fa3540d0d50 2024-03-27T06:14:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa354156790 con 0x7fa3540d0d50 2024-03-27T06:14:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c045070 con 0x7fa3540d0d50 2024-03-27T06:14:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.261+0000 7f3598c00640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7f3574074780 0x7f3574076ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.263+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== osd.2 v2:172.21.15.132:6823/1881883030 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3594071e50 con 0x7f3574074780 2024-03-27T06:14:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.262+0000 7f3e9d800640 1 -- 172.21.15.132:0/3307394720 >> 172.21.15.132:0/3307394720 conn(0x7f3e980ccbe0 msgr2=0x7f3e980ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.263+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c039050 con 0x7fa3540d0d50 2024-03-27T06:14:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.263+0000 7f3e9d800640 1 -- 172.21.15.132:0/3307394720 shutdown_connections 2024-03-27T06:14:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.264+0000 7f3e9d800640 1 -- 172.21.15.132:0/3307394720 wait complete. 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.263+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd2dc0043e0 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.264+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2ec068840 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.264+0000 7fd2e3e00640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd2cc06c580 0x7fd2cc06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.265+0000 7fd2eb400640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd2cc06c580 0x7fd2cc06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fd2eb400640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd2cc06c580 0x7fd2cc06ea40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd2dc057a60 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa32c001030 con 0x7fa3540d0d50 2024-03-27T06:14:12.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fd2e3e00640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7fd2cc0753a0 0x7fd2cc077840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7f3e9d800640 1 Processor -- start 2024-03-27T06:14:12.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2cc077f10 con 0x7fd2cc0753a0 2024-03-27T06:14:12.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd2dc0563f0 con 0x7fd2ec06c8a0 2024-03-27T06:14:12.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f556c04b060 con 0x7f557c06c8a0 2024-03-27T06:14:12.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f5572a00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c600 0x7f555c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f556c0adb10 con 0x7f557c06c8a0 2024-03-27T06:14:12.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f5572a00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f555c074820 0x7f555c076c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f555c0772f0 con 0x7f555c074820 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f556c0adf00 con 0x7f557c06c8a0 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.266+0000 7f3e9d800640 1 -- start start 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0039e0 0x7f3e980c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 0x7f3e981506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e9815e120 0x7f3e981ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9806d440 con 0x7f3e98158270 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9806d2c0 con 0x7f3e7c0039e0 2024-03-27T06:14:12.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.267+0000 7f3e9d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3e980cd700 con 0x7f3e9815e120 2024-03-27T06:14:12.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.268+0000 7fd2f1200640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7fd2cc0753a0 0x7fd2cc077840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.268+0000 7f3e96000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 0x7f3e981506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.268+0000 7fd2f1200640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7fd2cc0753a0 0x7fd2cc077840 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3599600640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f357406c580 0x7f357406ea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3588037000 tx=0x7f3588032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.268+0000 7f3e96000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 0x7f3e981506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37374/0 (socket says 172.21.15.132:37374) 2024-03-27T06:14:12.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 -- 172.21.15.132:0/2023072514 learned_addr learned my addr 172.21.15.132:0/2023072514 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96a00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0039e0 0x7f3e980c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f557a000640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c600 0x7f555c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f557b400640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f555c074820 0x7f555c076c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e97400640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e9815e120 0x7f3e981ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 -- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e9815e120 msgr2=0x7f3e981ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3e9815e120 0x7f3e981ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 -- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0039e0 msgr2=0x7f3e980c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0039e0 0x7f3e980c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96000640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9815bcd0 con 0x7f3e98158270 2024-03-27T06:14:12.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.270+0000 7f557a000640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c600 0x7f555c06eac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5568006fd0 tx=0x7f5568008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7f3e96a00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e7c0039e0 0x7f3e980c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:12.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.270+0000 7f557b400640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f555c074820 0x7f555c076c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.269+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa34c03e610 con 0x7fa3540d0d50 2024-03-27T06:14:12.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.270+0000 7f3e96000640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 0x7f3e981506c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400c410 tx=0x7f3e8400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa34b400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa33406c730 0x7fa33406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== osd.4 v2:172.21.15.134:6816/2337158882 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f555c0772f0 con 0x7f555c074820 2024-03-27T06:14:12.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84011790 con 0x7f3e98158270 2024-03-27T06:14:12.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e84017070 con 0x7f3e98158270 2024-03-27T06:14:12.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e980c5640 con 0x7f3e98158270 2024-03-27T06:14:12.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== osd.5 v2:172.21.15.134:6808/172932420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2cc077f10 con 0x7fd2cc0753a0 2024-03-27T06:14:12.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84011bb0 con 0x7f3e98158270 2024-03-27T06:14:12.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa34c0aea90 con 0x7fa3540d0d50 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa353400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa33406c730 0x7fa33406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa34b400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7fa334074930 0x7fa334076d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa34c030e50 con 0x7fa334074930 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa34c0aef20 con 0x7fa3540d0d50 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.271+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e980c5900 con 0x7f3e98158270 2024-03-27T06:14:12.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.272+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e9815dd00 con 0x7f3e98158270 2024-03-27T06:14:12.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.273+0000 7fa352a00640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7fa334074930 0x7fa334076d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.273+0000 7fa353400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa33406c730 0x7fa33406ebf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa344005880 tx=0x7fa344009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.274+0000 7fa352a00640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7fa334074930 0x7fa334076d50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.275+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 --> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e28002dd0 con 0x7f2e2c074730 2024-03-27T06:14:12.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.275+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== osd.6 v2:172.21.15.134:6800/3892608742 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa34c030e50 con 0x7fa334074930 2024-03-27T06:14:12.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.275+0000 7f2e48c00640 1 -- 172.21.15.132:0/949397829 <== osd.0 v2:172.21.15.132:6800/1970178383 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2e28002dd0 con 0x7f2e2c074730 2024-03-27T06:14:12.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.276+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7f2e2c074730 msgr2=0x7f2e2c076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.276+0000 7f2e52200640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7f2e2c074730 0x7f2e2c076b50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.277+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3e8401c460 con 0x7f3e98158270 2024-03-27T06:14:12.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.278+0000 7f3e8ea00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3e7806c600 0x7f3e7806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.278+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3e8408bfc0 con 0x7f3e98158270 2024-03-27T06:14:12.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f3e96a00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3e7806c600 0x7f3e7806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f3e8ea00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f3e78074820 0x7f3e78076c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e84014e70 con 0x7f3e78074820 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3e8408c3b0 con 0x7f3e98158270 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f3e97400640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f3e78074820 0x7f3e78076c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2e2c06c530 msgr2=0x7f2e2c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.279+0000 7f2e52200640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2e2c06c530 0x7f2e2c06e9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e38038000 tx=0x7f2e38008030 comp rx=0 tx=0).stop 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.280+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 msgr2=0x7f2e440e8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.280+0000 7f2e52200640 1 --2- 172.21.15.132:0/949397829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e44034390 0x7f2e440e8af0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d830 tx=0x7f2e4000dc30 comp rx=0 tx=0).stop 2024-03-27T06:14:12.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.280+0000 7f3e96a00640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3e7806c600 0x7f3e7806eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e88047000 tx=0x7f3e88006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.280+0000 7f2e51800640 1 -- 172.21.15.132:0/949397829 reap_dead start 2024-03-27T06:14:12.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.281+0000 7f3e97400640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f3e78074820 0x7f3e78076c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.281+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 shutdown_connections 2024-03-27T06:14:12.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.281+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 >> 172.21.15.132:0/949397829 conn(0x7f2e44050140 msgr2=0x7f2e44005ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.281+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 shutdown_connections 2024-03-27T06:14:12.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.281+0000 7f2e52200640 1 -- 172.21.15.132:0/949397829 wait complete. 2024-03-27T06:14:12.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.282+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== osd.7 v2:172.21.15.134:6824/789010038 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e84014e70 con 0x7f3e78074820 2024-03-27T06:14:12.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.290+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 --> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc8ec052620 con 0x7fc8d4074960 2024-03-27T06:14:12.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.291+0000 7fc8eaa00640 1 -- 172.21.15.132:0/1947215825 <== osd.3 v2:172.21.15.132:6808/2165878941 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc8ec052620 con 0x7fc8d4074960 2024-03-27T06:14:12.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.291+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fc8d4074960 msgr2=0x7fc8d4076d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.291+0000 7fc8f9c00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fc8d4074960 0x7fc8d4076d80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.292+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 --> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14dc04bc20 con 0x7f14c4074880 2024-03-27T06:14:12.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.292+0000 7f14e0c00640 1 -- 172.21.15.132:0/1486810240 <== osd.1 v2:172.21.15.132:6813/3659088073 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f14dc04bc20 con 0x7f14c4074880 2024-03-27T06:14:12.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc8d406c760 msgr2=0x7fc8d406ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc8d406c760 0x7fc8d406ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec04b330 tx=0x7fc8e0008040 comp rx=0 tx=0).stop 2024-03-27T06:14:12.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 msgr2=0x7fc8ec006f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 --2- 172.21.15.132:0/1947215825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc8d80039b0 0x7fc8ec006f10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4001970 tx=0x7fc8e4000bc0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f3400640 1 -- 172.21.15.132:0/1947215825 reap_dead start 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 shutdown_connections 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 >> 172.21.15.132:0/1947215825 conn(0x7fc8ec00e150 msgr2=0x7fc8ec0e0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.293+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 shutdown_connections 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.294+0000 7fc8f9c00640 1 -- 172.21.15.132:0/1947215825 wait complete. 2024-03-27T06:14:12.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.294+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 --> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2ec15b340 con 0x7fd2cc0753a0 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.294+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7f14c4074880 msgr2=0x7f14c4076ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.294+0000 7f14ea200640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7f14c4074880 0x7f14c4076ca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.294+0000 7fd2e3e00640 1 -- 172.21.15.132:0/2629381480 <== osd.5 v2:172.21.15.134:6808/172932420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd2ec15b340 con 0x7fd2cc0753a0 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f14c406c600 msgr2=0x7f14c406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f14c406c600 0x7f14c406eac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0d2a90 tx=0x7f14d0009040 comp rx=0 tx=0).stop 2024-03-27T06:14:12.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 msgr2=0x7f14dc1816f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 --2- 172.21.15.132:0/1486810240 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14dc00e260 0x7f14dc1816f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b410 tx=0x7f14d800b8d0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14e9800640 1 -- 172.21.15.132:0/1486810240 reap_dead start 2024-03-27T06:14:12.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 shutdown_connections 2024-03-27T06:14:12.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 >> 172.21.15.132:0/1486810240 conn(0x7f14dc050140 msgr2=0x7f14dc004c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 shutdown_connections 2024-03-27T06:14:12.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.295+0000 7f14ea200640 1 -- 172.21.15.132:0/1486810240 wait complete. 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7fd2cc0753a0 msgr2=0x7fd2cc077840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7fd2cc0753a0 0x7fd2cc077840 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd2cc06c580 msgr2=0x7fd2cc06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd2cc06c580 0x7fd2cc06ea40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).stop 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 msgr2=0x7fd2ec1f1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 --2- 172.21.15.132:0/2629381480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2ec06c8a0 0x7fd2ec1f1370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00ce10 tx=0x7fd2dc00a660 comp rx=0 tx=0).stop 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f1200640 1 -- 172.21.15.132:0/2629381480 reap_dead start 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 shutdown_connections 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 >> 172.21.15.132:0/2629381480 conn(0x7fd2ec0ccbe0 msgr2=0x7fd2ec0c99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 shutdown_connections 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.296+0000 7fd2f3200640 1 -- 172.21.15.132:0/2629381480 wait complete. 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.297+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 --> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f358c0cf270 con 0x7f3574074780 2024-03-27T06:14:12.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.298+0000 7f3591600640 1 -- 172.21.15.132:0/2358779695 <== osd.2 v2:172.21.15.132:6823/1881883030 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f358c0cf270 con 0x7f3574074780 2024-03-27T06:14:12.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.301+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7f3574074780 msgr2=0x7f3574076ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.301+0000 7f359c000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7f3574074780 0x7f3574076ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.302+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f357406c580 msgr2=0x7f357406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.302+0000 7f359c000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f357406c580 0x7f357406ea40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3588037000 tx=0x7f3588032000 comp rx=0 tx=0).stop 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.302+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 msgr2=0x7f358c0cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.302+0000 7f359c000640 1 --2- 172.21.15.132:0/2358779695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f358c0d51f0 0x7f358c0cd130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3594050b20 tx=0x7f3594069930 comp rx=0 tx=0).stop 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.303+0000 7f359a000640 1 -- 172.21.15.132:0/2358779695 reap_dead start 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.303+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 shutdown_connections 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.303+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 >> 172.21.15.132:0/2358779695 conn(0x7f358c050170 msgr2=0x7f358c0d3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.304+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 shutdown_connections 2024-03-27T06:14:12.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.305+0000 7f5581c00640 1 -- 172.21.15.132:0/302766146 --> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f557c068840 con 0x7f555c074820 2024-03-27T06:14:12.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.305+0000 7f359c000640 1 -- 172.21.15.132:0/2358779695 wait complete. 2024-03-27T06:14:12.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.305+0000 7f5572a00640 1 -- 172.21.15.132:0/302766146 <== osd.4 v2:172.21.15.134:6816/2337158882 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f557c068840 con 0x7f555c074820 2024-03-27T06:14:12.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.306+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f555c074820 msgr2=0x7f555c076c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.306+0000 7f5567e00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f555c074820 0x7f555c076c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.306+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c600 msgr2=0x7f555c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.306+0000 7f5567e00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c600 0x7f555c06eac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5568006fd0 tx=0x7f5568008040 comp rx=0 tx=0).stop 2024-03-27T06:14:12.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.307+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 msgr2=0x7f557c0c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.308+0000 7f5567e00640 1 --2- 172.21.15.132:0/302766146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c06c8a0 0x7f557c0c82e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f556c009990 tx=0x7f556c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.308+0000 7f557b400640 1 -- 172.21.15.132:0/302766146 reap_dead start 2024-03-27T06:14:12.336 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.337 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T06:14:12.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.309+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 shutdown_connections 2024-03-27T06:14:12.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.309+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 >> 172.21.15.132:0/302766146 conn(0x7f557c0ccbe0 msgr2=0x7f557c158ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.309+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 shutdown_connections 2024-03-27T06:14:12.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.309+0000 7f5567e00640 1 -- 172.21.15.132:0/302766146 wait complete. 2024-03-27T06:14:12.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.311+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 --> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa32c002dd0 con 0x7fa334074930 2024-03-27T06:14:12.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa34b400640 1 -- 172.21.15.132:0/1254595944 <== osd.6 v2:172.21.15.134:6800/3892608742 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa32c002dd0 con 0x7fa334074930 2024-03-27T06:14:12.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7fa334074930 msgr2=0x7fa334076d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7fa334074930 0x7fa334076d50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa33406c730 msgr2=0x7fa33406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa33406c730 0x7fa33406ebf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa344005880 tx=0x7fa344009040 comp rx=0 tx=0).stop 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 msgr2=0x7fa3540cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa35a400640 1 --2- 172.21.15.132:0/1254595944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3540d0d50 0x7fa3540cfc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009d90 tx=0x7fa34c004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.312+0000 7fa353e00640 1 -- 172.21.15.132:0/1254595944 reap_dead start 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.313+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 shutdown_connections 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.313+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 >> 172.21.15.132:0/1254595944 conn(0x7fa35406dbd0 msgr2=0x7fa3540c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.313+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 shutdown_connections 2024-03-27T06:14:12.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.313+0000 7fa35a400640 1 -- 172.21.15.132:0/1254595944 wait complete. 2024-03-27T06:14:12.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.314+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 --> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3e98075cc0 con 0x7f3e78074820 2024-03-27T06:14:12.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.314+0000 7f3e8ea00640 1 -- 172.21.15.132:0/2023072514 <== osd.7 v2:172.21.15.134:6824/789010038 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3e98075cc0 con 0x7f3e78074820 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f3e78074820 msgr2=0x7f3e78076c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f3e78074820 0x7f3e78076c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3e7806c600 msgr2=0x7f3e7806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3e7806c600 0x7f3e7806eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e88047000 tx=0x7f3e88006040 comp rx=0 tx=0).stop 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 msgr2=0x7f3e981506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e9d800640 1 --2- 172.21.15.132:0/2023072514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e98158270 0x7f3e981506c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400c410 tx=0x7f3e8400c810 comp rx=0 tx=0).stop 2024-03-27T06:14:12.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.315+0000 7f3e97400640 1 -- 172.21.15.132:0/2023072514 reap_dead start 2024-03-27T06:14:12.343 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.343 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T06:14:12.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.319+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 shutdown_connections 2024-03-27T06:14:12.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.319+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 >> 172.21.15.132:0/2023072514 conn(0x7f3e980ccbe0 msgr2=0x7f3e98158b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.319+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 shutdown_connections 2024-03-27T06:14:12.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.319+0000 7f3e9d800640 1 -- 172.21.15.132:0/2023072514 wait complete. 2024-03-27T06:14:12.345 INFO:teuthology.orchestra.run.smithi132.stdout:55834574855 2024-03-27T06:14:12.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T06:14:12.346 INFO:teuthology.orchestra.run.smithi132.stdout:55834574855 2024-03-27T06:14:12.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-27T06:14:12.347 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T06:14:12.348 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-27T06:14:12.349 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-27T06:14:12.351 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:12.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-27T06:14:12.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.686+0000 7f8223000640 1 Processor -- start 2024-03-27T06:14:12.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f459da00640 1 Processor -- start 2024-03-27T06:14:12.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.686+0000 7f8223000640 1 -- start start 2024-03-27T06:14:12.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f8223000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f821c06daf0 0x7f821c15f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f8223000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821c0c1120 con 0x7f821c158f10 2024-03-27T06:14:12.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f8223000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821c0c13e0 con 0x7f821c06b520 2024-03-27T06:14:12.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f8223000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f821c15fc20 con 0x7f821c06daf0 2024-03-27T06:14:12.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.689+0000 7f8221000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f821c06b520 0x7f821c06b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37746/0 (socket says 172.21.15.132:37746) 2024-03-27T06:14:12.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.689+0000 7f821b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f821c06daf0 0x7f821c15f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.688+0000 7f459da00640 1 -- start start 2024-03-27T06:14:12.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f821b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f821c06daf0 0x7f821c15f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37378/0 (socket says 172.21.15.132:37378) 2024-03-27T06:14:12.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f821b400640 1 -- 172.21.15.132:0/3543901466 learned_addr learned my addr 172.21.15.132:0/3543901466 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.689+0000 7f459da00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f45981572b0 0x7f45980b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.689+0000 7f459da00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f459da00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f459806cdd0 con 0x7f4598070de0 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f459da00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4598071d60 con 0x7f45981572b0 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f459da00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4598072040 con 0x7f45980c64d0 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f4597400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.690+0000 7f4597400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35206/0 (socket says 172.21.15.132:35206) 2024-03-27T06:14:12.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.691+0000 7f4597400640 1 -- 172.21.15.132:0/1257573665 learned_addr learned my addr 172.21.15.132:0/1257573665 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.691+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 667268585 0 0) 0x7f821c0c1120 con 0x7f821c158f10 2024-03-27T06:14:12.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.691+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8200003520 con 0x7f821c158f10 2024-03-27T06:14:12.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.691+0000 7f4597400640 1 -- 172.21.15.132:0/1257573665 >> v2:172.21.15.132:3300/0 conn(0x7f45981572b0 msgr2=0x7f45980b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.691+0000 7f4597400640 1 --2- 172.21.15.132:0/1257573665 >> v2:172.21.15.132:3300/0 conn(0x7f45981572b0 0x7f45980b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.692+0000 7f4597400640 1 -- 172.21.15.132:0/1257573665 >> v1:172.21.15.132:6789/0 conn(0x7f4598070de0 legacy=0x7f4598071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.692+0000 7f4597400640 1 -- 172.21.15.132:0/1257573665 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4598072320 con 0x7f45980c64d0 2024-03-27T06:14:12.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.692+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071649229 0 0) 0x7f821c0c13e0 con 0x7f821c06b520 2024-03-27T06:14:12.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.692+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f821c0c1120 con 0x7f821c06b520 2024-03-27T06:14:12.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.693+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 876281188 0 0) 0x7f8200003520 con 0x7f821c158f10 2024-03-27T06:14:12.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.693+0000 7f821b400640 1 -- 172.21.15.132:0/3543901466 >> v1:172.21.15.134:6789/0 conn(0x7f821c06b520 legacy=0x7f821c06b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.694+0000 7f821b400640 1 -- 172.21.15.132:0/3543901466 >> v1:172.21.15.132:6789/0 conn(0x7f821c158f10 legacy=0x7f821c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.694+0000 7f821b400640 1 -- 172.21.15.132:0/3543901466 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821c0c16c0 con 0x7f821c06daf0 2024-03-27T06:14:12.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.694+0000 7f4597400640 1 --2- 172.21.15.132:0/1257573665 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4584009870 tx=0x7f458402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fffed07c645bd236 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.695+0000 7f4596a00640 1 -- 172.21.15.132:0/1257573665 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584035020 con 0x7f45980c64d0 2024-03-27T06:14:12.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.695+0000 7f4596a00640 1 -- 172.21.15.132:0/1257573665 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4584031870 con 0x7f45980c64d0 2024-03-27T06:14:12.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.695+0000 7f4596a00640 1 -- 172.21.15.132:0/1257573665 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584031c80 con 0x7f45980c64d0 2024-03-27T06:14:12.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.696+0000 7f821b400640 1 --2- 172.21.15.132:0/3543901466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06daf0 0x7f821c15f6e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8214009e50 tx=0x7f821402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341c40c0854df1e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.696+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080015a0 con 0x7f821c06daf0 2024-03-27T06:14:12.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.696+0000 7f459da00640 1 -- 172.21.15.132:0/1257573665 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 msgr2=0x7f45980c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.696+0000 7f459da00640 1 --2- 172.21.15.132:0/1257573665 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4584009870 tx=0x7f458402b910 comp rx=0 tx=0).stop 2024-03-27T06:14:12.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821402c6e0 con 0x7f821c06daf0 2024-03-27T06:14:12.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f459da00640 1 -- 172.21.15.132:0/1257573665 shutdown_connections 2024-03-27T06:14:12.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f459da00640 1 --2- 172.21.15.132:0/1257573665 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45980c68f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f459da00640 1 --2- 172.21.15.132:0/1257573665 >> v2:172.21.15.132:3300/0 conn(0x7f45981572b0 0x7f45980b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f459da00640 1 -- 172.21.15.132:0/1257573665 >> 172.21.15.132:0/1257573665 conn(0x7f45980cc7a0 msgr2=0x7f45980c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f821aa00640 1 -- 172.21.15.132:0/3543901466 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0015f0 con 0x7f821c06daf0 2024-03-27T06:14:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f743d800640 1 Processor -- start 2024-03-27T06:14:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.698+0000 7f459da00640 1 -- 172.21.15.132:0/1257573665 shutdown_connections 2024-03-27T06:14:12.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.698+0000 7f8223000640 1 -- 172.21.15.132:0/3543901466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06daf0 msgr2=0x7f821c15f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.698+0000 7f8223000640 1 --2- 172.21.15.132:0/3543901466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06daf0 0x7f821c15f6e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8214009e50 tx=0x7f821402bdc0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.697+0000 7f743d800640 1 -- start start 2024-03-27T06:14:12.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.698+0000 7fad32800640 1 Processor -- start 2024-03-27T06:14:12.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.699+0000 7fad32800640 1 -- start start 2024-03-27T06:14:12.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.699+0000 7f8223000640 1 -- 172.21.15.132:0/3543901466 shutdown_connections 2024-03-27T06:14:12.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.698+0000 7f459da00640 1 -- 172.21.15.132:0/1257573665 wait complete. 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.699+0000 7f8223000640 1 --2- 172.21.15.132:0/3543901466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06daf0 0x7f821c15f6e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.699+0000 7f8223000640 1 -- 172.21.15.132:0/3543901466 >> 172.21.15.132:0/3543901466 conn(0x7f821c0cc060 msgr2=0x7f821c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f8223000640 1 -- 172.21.15.132:0/3543901466 shutdown_connections 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f8223000640 1 -- 172.21.15.132:0/3543901466 wait complete. 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f459da00640 1 Processor -- start 2024-03-27T06:14:12.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f459da00640 1 -- start start 2024-03-27T06:14:12.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f743d800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7438152c60 0x7f743806fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f743d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f743d800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743806f6c0 con 0x7f74380b0d10 2024-03-27T06:14:12.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f743d800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f743806f250 con 0x7f7438152c60 2024-03-27T06:14:12.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.700+0000 7f743d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f743806e640 con 0x7f7438159c30 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad32800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fad2c0d0d50 0x7fad2c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad32800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fad2c0cb440 0x7fad2c0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad32800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2c0cb9f0 con 0x7fad2c0d0970 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad32800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fad2c15e2e0 con 0x7fad2c0d0d50 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45981f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 0x7f45981f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45980c6e30 con 0x7f45981572b0 2024-03-27T06:14:12.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f45980728e0 con 0x7f4598070de0 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f459da00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4598071960 con 0x7f45980c64d0 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad32800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fad2c15e5c0 con 0x7fad2c0cb440 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad2b400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fad2c0d0970 0x7fad2c154aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33018/0 (socket says 172.21.15.132:33018) 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.701+0000 7fad2b400640 1 -- 172.21.15.132:0/2487182972 learned_addr learned my addr 172.21.15.132:0/2487182972 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f4597400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45981f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f4597e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f4597e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56286/0 (socket says 172.21.15.132:56286) 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f4597e00640 1 -- 172.21.15.132:0/3801605615 learned_addr learned my addr 172.21.15.132:0/3801605615 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f459d000640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 0x7f45981f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2856670970 0 0) 0x7fad2c0cb9f0 con 0x7fad2c0d0970 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad10003520 con 0x7fad2c0d0970 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7fd785a00640 1 Processor -- start 2024-03-27T06:14:12.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7fd785a00640 1 -- start start 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f7436000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f7436000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35218/0 (socket says 172.21.15.132:35218) 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.702+0000 7f7436000640 1 -- 172.21.15.132:0/1396986868 learned_addr learned my addr 172.21.15.132:0/1396986868 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 596730425 0 0) 0x7fad10003520 con 0x7fad2c0d0970 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 >> v2:172.21.15.134:3300/0 conn(0x7fad2c0cb440 msgr2=0x7fad2c0c8a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 --2- 172.21.15.132:0/2487182972 >> v2:172.21.15.134:3300/0 conn(0x7fad2c0cb440 0x7fad2c0c8a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 >> v2:172.21.15.132:3300/0 conn(0x7fad2c0d0d50 msgr2=0x7fad2c15b6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 --2- 172.21.15.132:0/2487182972 >> v2:172.21.15.132:3300/0 conn(0x7fad2c0d0d50 0x7fad2c15b6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c15e8a0 con 0x7fad2c0d0970 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7436a00640 1 --2- 172.21.15.132:0/1396986868 >> v2:172.21.15.132:3300/0 conn(0x7f7438152c60 0x7f743806fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f4597e00640 1 -- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 msgr2=0x7f45981f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f4597e00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45981f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7436000640 1 -- 172.21.15.132:0/1396986868 >> v2:172.21.15.132:3300/0 conn(0x7f7438152c60 msgr2=0x7f743806fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7436000640 1 --2- 172.21.15.132:0/1396986868 >> v2:172.21.15.132:3300/0 conn(0x7f7438152c60 0x7f743806fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7f8223000640 1 Processor -- start 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f4597e00640 1 -- 172.21.15.132:0/3801605615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 msgr2=0x7f45981f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f4597e00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 0x7f45981f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f4597e00640 1 -- 172.21.15.132:0/3801605615 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4598157ad0 con 0x7f4598070de0 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7f459d000640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 0x7f45981f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7f4597e00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f458c00b4d0 tx=0x7f458c00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7f9cc3200640 1 Processor -- start 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7436000640 1 -- 172.21.15.132:0/1396986868 >> v1:172.21.15.134:6789/0 conn(0x7f74380b0d10 legacy=0x7f74380ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7436000640 1 -- 172.21.15.132:0/1396986868 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74380c54f0 con 0x7f7438159c30 2024-03-27T06:14:12.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7f7435600640 1 -- 172.21.15.132:0/1396986868 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338045603 0 0) 0x7f743806f6c0 con 0x7f74380b0d10 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4217832196 0 0) 0x7fad14001670 con 0x7fad2c0d0970 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 >> v1:172.21.15.132:6789/0 conn(0x7fad2c0d0970 legacy=0x7fad2c154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10003a00 0x7fad10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 0x7fad10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad100053c0 0x7fad1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c15e2e0 con 0x7fad10003a00 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c15e5c0 con 0x7fad10004800 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.703+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fad1000c2b0 con 0x7fad100053c0 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2be00640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad100053c0 0x7fad1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2aa00640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 0x7fad10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2b400640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10003a00 0x7fad10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7f8223000640 1 -- start start 2024-03-27T06:14:12.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2aa00640 1 -- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad100053c0 msgr2=0x7fad1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2aa00640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad100053c0 0x7fad1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fad2aa00640 1 -- 172.21.15.132:0/2487182972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10003a00 msgr2=0x7fad10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7f9cc3200640 1 -- start start 2024-03-27T06:14:12.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.704+0000 7fd785a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c014070 con 0x7f4598070de0 2024-03-27T06:14:12.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458c010610 con 0x7f4598070de0 2024-03-27T06:14:12.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c010a00 con 0x7f4598070de0 2024-03-27T06:14:12.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7f7436000640 1 --2- 172.21.15.132:0/1396986868 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7424009870 tx=0x7f742402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6adad0f1dc2df1ce server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd785a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780376e0 con 0x7fd7780d92c0 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd785a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780502d0 con 0x7fd778037180 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd785a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd7780505b0 con 0x7fd778038d60 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd77f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd77f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35250/0 (socket says 172.21.15.132:35250) 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fd77f400640 1 -- 172.21.15.132:0/1881644498 learned_addr learned my addr 172.21.15.132:0/1881644498 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7fd77ea00640 1 -- 172.21.15.132:0/1881644498 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2212500179 0 0) 0x7fd7780502d0 con 0x7fd778037180 2024-03-27T06:14:12.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7fd77f400640 1 -- 172.21.15.132:0/1881644498 >> v1:172.21.15.134:6789/0 conn(0x7fd778037180 legacy=0x7fd7780e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7fd77f400640 1 -- 172.21.15.132:0/1881644498 >> v1:172.21.15.132:6789/0 conn(0x7fd7780d92c0 legacy=0x7fd778038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7fd77f400640 1 -- 172.21.15.132:0/1881644498 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd778050890 con 0x7fd778038d60 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f9cc3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 0x7f9cbc0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f9cc3200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9cbc15e320 0x7f9cbc15e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fad2aa00640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10003a00 0x7fad10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fad2aa00640 1 -- 172.21.15.132:0/2487182972 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad10003520 con 0x7fad10004800 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.705+0000 7fad2b400640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10003a00 0x7fad10003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7fad2be00640 1 -- 172.21.15.132:0/2487182972 reap_dead start 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f9cc3200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc06cb60 con 0x7f9cbc0c12e0 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f9cc3200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f9cbc0c2580 con 0x7f9cbc15e320 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f9cc3200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc0c27d0 con 0x7f9cbc15df40 2024-03-27T06:14:12.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f7435600640 1 -- 172.21.15.132:0/1396986868 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424035020 con 0x7f7438159c30 2024-03-27T06:14:12.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.706+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459815a150 con 0x7f4598070de0 2024-03-27T06:14:12.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459815a6d0 con 0x7f4598070de0 2024-03-27T06:14:12.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7fad2aa00640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 0x7fad10004c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0097d0 tx=0x7fad10005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7b14958d1d239d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f7435600640 1 -- 172.21.15.132:0/1396986868 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7424031640 con 0x7f7438159c30 2024-03-27T06:14:12.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f7435600640 1 -- 172.21.15.132:0/1396986868 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742402d040 con 0x7f7438159c30 2024-03-27T06:14:12.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc1e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 0x7f9cbc0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc2800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9cbc15e320 0x7f9cbc15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc1e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 0x7f9cbc0c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35254/0 (socket says 172.21.15.132:35254) 2024-03-27T06:14:12.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc1e00640 1 -- 172.21.15.132:0/2151196089 learned_addr learned my addr 172.21.15.132:0/2151196089 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c037070 con 0x7fad10004800 2024-03-27T06:14:12.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1c032600 con 0x7fad10004800 2024-03-27T06:14:12.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7fad2a000640 1 -- 172.21.15.132:0/2487182972 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0093d0 con 0x7fad10004800 2024-03-27T06:14:12.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 -- 172.21.15.132:0/1396986868 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 msgr2=0x7f74380c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 --2- 172.21.15.132:0/1396986868 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7424009870 tx=0x7f742402b8f0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc2800640 1 -- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 msgr2=0x7f9cbc0c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:12.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc2800640 1 --2- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 0x7f9cbc0c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc2800640 1 -- 172.21.15.132:0/2151196089 >> v1:172.21.15.132:6789/0 conn(0x7f9cbc0c12e0 legacy=0x7f9cbc0c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.707+0000 7f9cc2800640 1 -- 172.21.15.132:0/2151196089 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc0c2ab0 con 0x7f9cbc15e320 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 -- 172.21.15.132:0/1396986868 shutdown_connections 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 --2- 172.21.15.132:0/1396986868 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f74380c4fb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 --2- 172.21.15.132:0/1396986868 >> v2:172.21.15.132:3300/0 conn(0x7f7438152c60 0x7f743806fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f743d800640 1 -- 172.21.15.132:0/1396986868 >> 172.21.15.132:0/1396986868 conn(0x7f7438067280 msgr2=0x7f7438067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f8223000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 0x7f821c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f8223000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.708+0000 7f8223000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 0x7f821c071470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f821b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f821b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35266/0 (socket says 172.21.15.132:35266) 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f8223000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f821c0c1f60 con 0x7f821c06b520 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f8223000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f821c0c1c80 con 0x7f821c158f10 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f821c06cec0 con 0x7f821c06daf0 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f821b400640 1 -- 172.21.15.132:0/1525359274 learned_addr learned my addr 172.21.15.132:0/1525359274 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7fad32800640 1 -- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 msgr2=0x7fad10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f743d800640 1 -- 172.21.15.132:0/1396986868 shutdown_connections 2024-03-27T06:14:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f8221000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 0x7f821c071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7fad32800640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 0x7fad10004c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0097d0 tx=0x7fad10005830 comp rx=0 tx=0).stop 2024-03-27T06:14:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f821be00640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 0x7f821c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7fd77f400640 1 --2- 172.21.15.132:0/1881644498 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0098a0 tx=0x7fd76c02ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff1e6fbdd1fe5ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f459806f3e0 con 0x7f4598070de0 2024-03-27T06:14:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fad32800640 1 -- 172.21.15.132:0/2487182972 shutdown_connections 2024-03-27T06:14:12.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fad32800640 1 --2- 172.21.15.132:0/2487182972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad10004800 0x7fad10004c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fad32800640 1 -- 172.21.15.132:0/2487182972 >> 172.21.15.132:0/2487182972 conn(0x7fad2c06dbd0 msgr2=0x7fad2c06dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fd77ea00640 1 -- 172.21.15.132:0/1881644498 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c02f850 con 0x7fd778038d60 2024-03-27T06:14:12.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821b400640 1 -- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 msgr2=0x7f821c071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fad32800640 1 -- 172.21.15.132:0/2487182972 shutdown_connections 2024-03-27T06:14:12.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fad32800640 1 -- 172.21.15.132:0/2487182972 wait complete. 2024-03-27T06:14:12.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fd77ea00640 1 -- 172.21.15.132:0/1881644498 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c02f9f0 con 0x7fd778038d60 2024-03-27T06:14:12.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7fd77ea00640 1 -- 172.21.15.132:0/1881644498 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c02fdc0 con 0x7fd778038d60 2024-03-27T06:14:12.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821b400640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 0x7f821c071470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.709+0000 7f743d800640 1 -- 172.21.15.132:0/1396986868 wait complete. 2024-03-27T06:14:12.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821b400640 1 -- 172.21.15.132:0/1525359274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 msgr2=0x7f821c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821b400640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 0x7f821c15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821b400640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821c156de0 con 0x7f821c06daf0 2024-03-27T06:14:12.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f821be00640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 0x7f821c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.710+0000 7f8221000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 0x7f821c071470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f821b400640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8214009e50 tx=0x7f8214004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f9cc2800640 1 --2- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15e790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0097a0 tx=0x7f9cac02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df80df9f8db483f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214043020 con 0x7f821c06daf0 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8214007930 con 0x7f821c06daf0 2024-03-27T06:14:12.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214007d00 con 0x7f821c06daf0 2024-03-27T06:14:12.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7f9cc0a00640 1 -- 172.21.15.132:0/2151196089 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac034070 con 0x7f9cbc15e320 2024-03-27T06:14:12.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7fd785a00640 1 -- 172.21.15.132:0/1881644498 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 msgr2=0x7fd7780e2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.711+0000 7fd785a00640 1 --2- 172.21.15.132:0/1881644498 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0098a0 tx=0x7fd76c02ba30 comp rx=0 tx=0).stop 2024-03-27T06:14:12.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7f9cc0a00640 1 -- 172.21.15.132:0/2151196089 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cac030800 con 0x7f9cbc15e320 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7f9cc0a00640 1 -- 172.21.15.132:0/2151196089 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac030bd0 con 0x7f9cbc15e320 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7fd785a00640 1 -- 172.21.15.132:0/1881644498 shutdown_connections 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7fd785a00640 1 --2- 172.21.15.132:0/1881644498 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778038d60 0x7fd7780e2310 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7fd785a00640 1 -- 172.21.15.132:0/1881644498 >> 172.21.15.132:0/1881644498 conn(0x7fd778054610 msgr2=0x7fd778054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821c159770 con 0x7f821c06daf0 2024-03-27T06:14:12.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f821c159d30 con 0x7f821c06daf0 2024-03-27T06:14:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f458c017070 con 0x7f4598070de0 2024-03-27T06:14:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7fd785a00640 1 -- 172.21.15.132:0/1881644498 shutdown_connections 2024-03-27T06:14:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.712+0000 7fd785a00640 1 -- 172.21.15.132:0/1881644498 wait complete. 2024-03-27T06:14:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 -- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15e320 msgr2=0x7f9cbc15e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 --2- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15e790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0097a0 tx=0x7f9cac02b940 comp rx=0 tx=0).stop 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 -- 172.21.15.132:0/2151196089 shutdown_connections 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 --2- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15e790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 --2- 172.21.15.132:0/2151196089 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc15df40 0x7f9cbc0c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f9cc3200640 1 -- 172.21.15.132:0/2151196089 >> 172.21.15.132:0/2151196089 conn(0x7f9cbc06e790 msgr2=0x7f9cbc06eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f458be00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 0x7f457806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f458c08b080 con 0x7f4598070de0 2024-03-27T06:14:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7fad32800640 1 Processor -- start 2024-03-27T06:14:12.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7fad32800640 1 -- start start 2024-03-27T06:14:12.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f9cc3200640 1 -- 172.21.15.132:0/2151196089 shutdown_connections 2024-03-27T06:14:12.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f743d800640 1 Processor -- start 2024-03-27T06:14:12.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f4597400640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 0x7f457806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.713+0000 7f743d800640 1 -- start start 2024-03-27T06:14:12.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 0x7f74380cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f9cc3200640 1 -- 172.21.15.132:0/2151196089 wait complete. 2024-03-27T06:14:12.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 0x7f74381566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fd785a00640 1 Processor -- start 2024-03-27T06:14:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821c068840 con 0x7f821c06daf0 2024-03-27T06:14:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f7438145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f743815cac0 con 0x7f7438152c60 2024-03-27T06:14:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f743815c7e0 con 0x7f74380b0d10 2024-03-27T06:14:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.714+0000 7f743d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f743806eee0 con 0x7f7438159c30 2024-03-27T06:14:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f9b6b600640 1 Processor -- start 2024-03-27T06:14:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f9b6b600640 1 -- start start 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 0x7f74381566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 0x7f74380cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 0x7f74381566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37420/0 (socket says 172.21.15.132:37420) 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436000640 1 -- 172.21.15.132:0/4265245499 learned_addr learned my addr 172.21.15.132:0/4265245499 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7437400640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f7438145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 -- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 msgr2=0x7f7438145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f7438145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 -- 172.21.15.132:0/4265245499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 msgr2=0x7f74381566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 0x7f74381566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7f7436a00640 1 -- 172.21.15.132:0/4265245499 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7438147800 con 0x7f74380b0d10 2024-03-27T06:14:12.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7f7436a00640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 0x7f74380cad50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f742800d550 tx=0x7f742800da10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad2c0b5fa0 0x7fad2c15d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 0x7fad2c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c0cc970 con 0x7fad10004800 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c0cc7f0 con 0x7fad2c15d5b0 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fad32800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fad2c15b6d0 con 0x7fad2c0b5fa0 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7fad2b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7fad2b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37426/0 (socket says 172.21.15.132:37426) 2024-03-27T06:14:12.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7fad2b400640 1 -- 172.21.15.132:0/3445426554 learned_addr learned my addr 172.21.15.132:0/3445426554 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7f4597400640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 0x7f457806ea40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4584036000 tx=0x7f458403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.715+0000 7fd785a00640 1 -- start start 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.716+0000 7fad2be00640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 0x7fad2c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fad2b400640 1 -- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad2c0b5fa0 msgr2=0x7fad2c15d070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fad2b400640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad2c0b5fa0 0x7fad2c15d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280125f0 con 0x7f74380b0d10 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7428019070 con 0x7f74380b0d10 2024-03-27T06:14:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fd785a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8214046020 con 0x7f821c06daf0 2024-03-27T06:14:12.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280129b0 con 0x7f74380b0d10 2024-03-27T06:14:12.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fd785a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd778038d60 0x7fd7780d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fd785a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 0x7fd778052bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7f9cc3200640 1 Processor -- start 2024-03-27T06:14:12.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f8213e00640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 0x7f81fc06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fad2b400640 1 -- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 msgr2=0x7fad2c156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fad2b400640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 0x7fad2c156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.717+0000 7fad2b400640 1 -- 172.21.15.132:0/3445426554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c150490 con 0x7fad10004800 2024-03-27T06:14:12.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9cc3200640 1 -- start start 2024-03-27T06:14:12.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74380c92a0 con 0x7f74380b0d10 2024-03-27T06:14:12.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74380c9860 con 0x7f74380b0d10 2024-03-27T06:14:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f82140ad6e0 con 0x7f821c06daf0 2024-03-27T06:14:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f821be00640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 0x7f81fc06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7fad2be00640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 0x7fad2c156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35290/0 (socket says 172.21.15.132:35290) 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b6b600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 0x7f9b640c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b6b600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b64157a60 con 0x7f9b6406b520 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b6b600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6406d110 con 0x7f9b640c8160 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b6b600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9b640c1790 con 0x7f9b6406b940 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7fd785a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd7780e2f60 con 0x7fd7780d92c0 2024-03-27T06:14:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fd785a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd778038130 con 0x7fd778038d60 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b68c00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9b640c8160 0x7f9b64150df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37776/0 (socket says 172.21.15.132:37776) 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f9b68c00640 1 -- 172.21.15.132:0/2279965764 learned_addr learned my addr 172.21.15.132:0/2279965764 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fd785a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd7780e30e0 con 0x7fd778037180 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7fd785000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 0x7fd778052bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fd77f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd778038d60 0x7fd7780d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fd77fe00640 1 -- 172.21.15.132:0/1313719550 learned_addr learned my addr 172.21.15.132:0/1313719550 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fad2b400640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad140017e0 tx=0x7fad140077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7f9b69600640 1 --2- 172.21.15.132:0/2279965764 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 0x7f9b640c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.718+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f459806f3e0 con 0x7f4598070de0 2024-03-27T06:14:12.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7f821be00640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 0x7f81fc06eaf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82080012b0 tx=0x7f8208009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401a070 con 0x7fad10004800 2024-03-27T06:14:12.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1400fae0 con 0x7fad10004800 2024-03-27T06:14:12.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1400fe90 con 0x7fad10004800 2024-03-27T06:14:12.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c1f5430 con 0x7fad10004800 2024-03-27T06:14:12.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.719+0000 7f9b63400640 1 -- 172.21.15.132:0/2279965764 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870880044 0 0) 0x7f9b6406d110 con 0x7f9b640c8160 2024-03-27T06:14:12.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7f9b63400640 1 -- 172.21.15.132:0/2279965764 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b4c003520 con 0x7f9b640c8160 2024-03-27T06:14:12.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7f021a800640 1 Processor -- start 2024-03-27T06:14:12.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fd77fe00640 1 -- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd778038d60 msgr2=0x7fd7780d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fd77fe00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd778038d60 0x7fd7780d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fd77fe00640 1 -- 172.21.15.132:0/1313719550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 msgr2=0x7fd778052bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fd77fe00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 0x7fd778052bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c155970 con 0x7fad10004800 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc0c12e0 0x7f9cbc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 0x7f9cbc200d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f021a800640 1 -- start start 2024-03-27T06:14:12.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7f9b69600640 1 -- 172.21.15.132:0/2279965764 >> v1:172.21.15.134:6789/0 conn(0x7f9b640c8160 legacy=0x7f9b64150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7f9b69600640 1 -- 172.21.15.132:0/2279965764 >> v1:172.21.15.132:6789/0 conn(0x7f9b6406b520 legacy=0x7f9b640c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7f9b69600640 1 -- 172.21.15.132:0/2279965764 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6406d3f0 con 0x7f9b6406b940 2024-03-27T06:14:12.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7428012b50 con 0x7f74380b0d10 2024-03-27T06:14:12.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc3200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 0x7f9cbc200d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 0x7f9cbc200d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56358/0 (socket says 172.21.15.132:56358) 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.721+0000 7f9cc1400640 1 -- 172.21.15.132:0/3434815371 learned_addr learned my addr 172.21.15.132:0/3434815371 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f9cc2800640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc0c79c0 con 0x7f9cbc15e320 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc0c6d90 con 0x7f9cbc15df40 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc154540 con 0x7f9cbc0c12e0 2024-03-27T06:14:12.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f742ea00640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 0x7f741806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f9cc1e00640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc0c12e0 0x7f9cbc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f021a800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f02140c8a30 0x7f0214154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f742808b810 con 0x7f74380b0d10 2024-03-27T06:14:12.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7438068840 con 0x7f74380b0d10 2024-03-27T06:14:12.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f7436000640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 0x7f741806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f021a800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02140cdad0 con 0x7f021406cc80 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f021a800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f021415b690 con 0x7f021406c8a0 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f021a800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f02140cc210 con 0x7f02140c8a30 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f0213400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f021406c8a0 0x7f02140c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37784/0 (socket says 172.21.15.132:37784) 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f0213400640 1 -- 172.21.15.132:0/3243459558 learned_addr learned my addr 172.21.15.132:0/3243459558 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f9b69600640 1 --2- 172.21.15.132:0/2279965764 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 0x7f9b640c6a60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009dc0 tx=0x7f9b5c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6b006de6268a066 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.720+0000 7fd77fe00640 1 -- 172.21.15.132:0/1313719550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77804dbb0 con 0x7fd778037180 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f821404d070 con 0x7f821c06daf0 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.722+0000 7fd77fe00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c3f0 tx=0x7fd774001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774017070 con 0x7fd778037180 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77804c6e0 con 0x7fd778037180 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77804df40 con 0x7fd778037180 2024-03-27T06:14:12.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7fd785000640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 0x7fd778052bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7f9b63400640 1 -- 172.21.15.132:0/2279965764 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c039070 con 0x7f9b6406b940 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.724+0000 7f9b63400640 1 -- 172.21.15.132:0/2279965764 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c02ccc0 con 0x7f9b6406b940 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.724+0000 7f9b63400640 1 -- 172.21.15.132:0/2279965764 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c034660 con 0x7f9b6406b940 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.724+0000 7f7436000640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 0x7f741806ea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74380c8810 tx=0x7f742403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.723+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fad1401d070 con 0x7fad10004800 2024-03-27T06:14:12.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.724+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2c1f5190 con 0x7fad10004800 2024-03-27T06:14:12.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.724+0000 7f0213e00640 1 --2- 172.21.15.132:0/3243459558 >> v2:172.21.15.132:3300/0 conn(0x7f02140c8a30 0x7f0214154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 -- 172.21.15.132:0/2279965764 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 msgr2=0x7f9b640c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 --2- 172.21.15.132:0/2279965764 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 0x7f9b640c6a60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009dc0 tx=0x7f9b5c02bf10 comp rx=0 tx=0).stop 2024-03-27T06:14:12.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f0213e00640 1 -- 172.21.15.132:0/3243459558 >> v1:172.21.15.134:6789/0 conn(0x7f021406c8a0 legacy=0x7f02140c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f0213e00640 1 -- 172.21.15.132:0/3243459558 >> v1:172.21.15.132:6789/0 conn(0x7f021406cc80 legacy=0x7f0214154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f0213e00640 1 -- 172.21.15.132:0/3243459558 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021415b970 con 0x7f02140c8a30 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7fad23400640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 0x7fad0806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fad1408b270 con 0x7fad10004800 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 -- 172.21.15.132:0/2279965764 shutdown_connections 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 --2- 172.21.15.132:0/2279965764 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b940 0x7f9b640c6a60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 -- 172.21.15.132:0/2279965764 >> 172.21.15.132:0/2279965764 conn(0x7f9b640cc060 msgr2=0x7f9b640cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7fad2aa00640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 0x7fad0806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9b6b600640 1 -- 172.21.15.132:0/2279965764 shutdown_connections 2024-03-27T06:14:12.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7740035f0 con 0x7fd778037180 2024-03-27T06:14:12.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774017070 con 0x7fd778037180 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9cc2800640 1 -- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc0c12e0 msgr2=0x7f9cbc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9cc2800640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc0c12e0 0x7f9cbc06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9cc2800640 1 -- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 msgr2=0x7f9cbc200d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9cc2800640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 0x7f9cbc200d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.725+0000 7f9cc2800640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc15b360 con 0x7f9cbc15e320 2024-03-27T06:14:12.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.726+0000 7fad2aa00640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 0x7fad0806ec60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0d1160 tx=0x7fad1c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.726+0000 7f9cc2800640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15cc30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0708f0 tx=0x7f9cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.727+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75c005290 con 0x7fd778037180 2024-03-27T06:14:12.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.726+0000 7f9b6b600640 1 -- 172.21.15.132:0/2279965764 wait complete. 2024-03-27T06:14:12.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.728+0000 7f0213e00640 1 --2- 172.21.15.132:0/3243459558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02140c8a30 0x7f0214154fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f020c009d90 tx=0x7f020c02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc59fcf5bea1feb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f7428059fa0 con 0x7f74380b0d10 2024-03-27T06:14:12.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7f9b6b600640 1 Processor -- start 2024-03-27T06:14:12.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.728+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd77401a070 con 0x7fd778037180 2024-03-27T06:14:12.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7fd773e00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 0x7fd76006ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd77408a3d0 con 0x7fd778037180 2024-03-27T06:14:12.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7f9b6b600640 1 -- start start 2024-03-27T06:14:12.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.729+0000 7fd77f400640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 0x7fd76006ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.730+0000 7fd77f400640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 0x7fd76006ebf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd76c036000 tx=0x7fd76c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 0x7f9b64146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b640c8160 0x7f9b641470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6406cec0 con 0x7f9b6406b940 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6406da40 con 0x7f9b640c8160 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b6b600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9b640c8540 con 0x7f9b6406b520 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b68c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b68c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35328/0 (socket says 172.21.15.132:35328) 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b68c00640 1 -- 172.21.15.132:0/835814704 learned_addr learned my addr 172.21.15.132:0/835814704 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.731+0000 7f9b63e00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 0x7f9b64146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 -- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b640c8160 msgr2=0x7f9b641470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b640c8160 0x7f9b641470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 -- 172.21.15.132:0/835814704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 msgr2=0x7f9b64146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 0x7f9b64146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 -- 172.21.15.132:0/835814704 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b640c5770 con 0x7f9b6406b520 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b63e00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 0x7f9b64146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.732+0000 7f9b68c00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b50001b40 tx=0x7f9b50014420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.733+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50001cf0 con 0x7f9b6406b520 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.733+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b50014cc0 con 0x7f9b6406b520 2024-03-27T06:14:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.733+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50005ad0 con 0x7f9b6406b520 2024-03-27T06:14:12.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.734+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac034040 con 0x7f9cbc15e320 2024-03-27T06:14:12.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.734+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cac03ac20 con 0x7f9cbc15e320 2024-03-27T06:14:12.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.734+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac0443f0 con 0x7f9cbc15e320 2024-03-27T06:14:12.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.734+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b640c3e50 con 0x7f9b6406b520 2024-03-27T06:14:12.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b640c51b0 con 0x7f9b6406b520 2024-03-27T06:14:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7f0212000640 1 -- 172.21.15.132:0/3243459558 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c039070 con 0x7f02140c8a30 2024-03-27T06:14:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7f0212000640 1 -- 172.21.15.132:0/3243459558 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c02ca30 con 0x7f02140c8a30 2024-03-27T06:14:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7f0212000640 1 -- 172.21.15.132:0/3243459558 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c02ce00 con 0x7f02140c8a30 2024-03-27T06:14:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fad140592b0 con 0x7fad10004800 2024-03-27T06:14:12.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.735+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd774058430 con 0x7fd778037180 2024-03-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.736+0000 7f021a800640 1 -- 172.21.15.132:0/3243459558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02140c8a30 msgr2=0x7f0214154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.736+0000 7f021a800640 1 --2- 172.21.15.132:0/3243459558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02140c8a30 0x7f0214154fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f020c009d90 tx=0x7f020c02bd00 comp rx=0 tx=0).stop 2024-03-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.736+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6406ded0 con 0x7f9b6406b520 2024-03-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.736+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cbc0c35e0 con 0x7f9cbc15e320 2024-03-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.736+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cbc0c3ba0 con 0x7f9cbc15e320 2024-03-27T06:14:12.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.737+0000 7f021a800640 1 -- 172.21.15.132:0/3243459558 shutdown_connections 2024-03-27T06:14:12.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.737+0000 7f021a800640 1 --2- 172.21.15.132:0/3243459558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02140c8a30 0x7f0214154fa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.737+0000 7f021a800640 1 -- 172.21.15.132:0/3243459558 >> 172.21.15.132:0/3243459558 conn(0x7f02140ccbe0 msgr2=0x7f02140ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:12.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.738+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cbc0686c0 con 0x7f9cbc15e320 2024-03-27T06:14:12.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.739+0000 7f021a800640 1 -- 172.21.15.132:0/3243459558 shutdown_connections 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.740+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9cac04b060 con 0x7f9cbc15e320 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.740+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9b50005c70 con 0x7f9b6406b520 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.741+0000 7f9b60c00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 0x7f9b4406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.741+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9b500935b0 con 0x7f9b6406b520 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.742+0000 7f9b63e00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 0x7f9b4406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.743+0000 7f9b63e00640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 0x7f9b4406e9f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b641512f0 tx=0x7f9b58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.742+0000 7f021a800640 1 -- 172.21.15.132:0/3243459558 wait complete. 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.744+0000 7f021a800640 1 Processor -- start 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.745+0000 7f021a800640 1 -- start start 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.742+0000 7f9cb6000640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 0x7f9ca006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.742+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9cac0ad8a0 con 0x7f9cbc15e320 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.743+0000 7f9cc1e00640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 0x7f9ca006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.744+0000 7f9cc1e00640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 0x7f9ca006ea20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009b60 tx=0x7f9cb0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f021406c8a0 0x7f0214153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 0x7f021414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9b6406ded0 con 0x7f9b6406b520 2024-03-27T06:14:12.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 0x7f021414c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f021406d440 con 0x7f021406c8a0 2024-03-27T06:14:12.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f021406d2c0 con 0x7f021406cc80 2024-03-27T06:14:12.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.746+0000 7f021a800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f02140cd700 con 0x7f02140c8a30 2024-03-27T06:14:12.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0212a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 0x7f021414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0212a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 0x7f021414bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56388/0 (socket says 172.21.15.132:56388) 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0212a00640 1 -- 172.21.15.132:0/2752829644 learned_addr learned my addr 172.21.15.132:0/2752829644 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 0x7f021414c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 -- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 msgr2=0x7f021414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 0x7f021414bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 -- 172.21.15.132:0/2752829644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f021406c8a0 msgr2=0x7f0214153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f021406c8a0 0x7f0214153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.747+0000 7f0213e00640 1 -- 172.21.15.132:0/2752829644 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02141e6ca0 con 0x7f02140c8a30 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.748+0000 7f0213e00640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 0x7f021414c400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f021406d6b0 tx=0x7f020c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.748+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c044020 con 0x7f02140c8a30 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.748+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c035cc0 con 0x7f02140c8a30 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.748+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c039040 con 0x7f02140c8a30 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.749+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0214152f60 con 0x7f02140c8a30 2024-03-27T06:14:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.749+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0214158e40 con 0x7f02140c8a30 2024-03-27T06:14:12.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.750+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9cac03f070 con 0x7f9cbc15e320 2024-03-27T06:14:12.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.750+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0214068840 con 0x7f02140c8a30 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.752+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f020c047020 con 0x7f02140c8a30 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.754+0000 7f020b400640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 0x7f01f406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.754+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f020c0addb0 con 0x7f02140c8a30 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.756+0000 7f0213400640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 0x7f01f406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.756+0000 7f0213400640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 0x7f01f406eac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0200000f60 tx=0x7f0200007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:12.758+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f020c04e030 con 0x7f02140c8a30 2024-03-27T06:14:13.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.142+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f821c06c570 con 0x7f821c06daf0 2024-03-27T06:14:13.133 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.143+0000 7f8213e00640 1 -- 172.21.15.132:0/1525359274 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f821407bd10 con 0x7f821c06daf0 2024-03-27T06:14:13.134 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.146+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 msgr2=0x7f81fc06eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.146+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 0x7f81fc06eaf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82080012b0 tx=0x7f8208009040 comp rx=0 tx=0).stop 2024-03-27T06:14:13.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.147+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 msgr2=0x7f821c1568a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.147+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8214009e50 tx=0x7f8214004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:13.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.147+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 shutdown_connections 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.147+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f81fc06c630 0x7f81fc06eaf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c158f10 0x7f821c071470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f821c06daf0 0x7f821c1568a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 --2- 172.21.15.132:0/1525359274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f821c06b520 0x7f821c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 >> 172.21.15.132:0/1525359274 conn(0x7f821c0cc060 msgr2=0x7f821c15f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 shutdown_connections 2024-03-27T06:14:13.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.148+0000 7f8223000640 1 -- 172.21.15.132:0/1525359274 wait complete. 2024-03-27T06:14:13.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.159+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f45980690d0 con 0x7f4598070de0 2024-03-27T06:14:13.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.160+0000 7f458be00640 1 -- 172.21.15.132:0/3801605615 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f45980690d0 con 0x7f4598070de0 2024-03-27T06:14:13.154 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.163+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f743806f8f0 con 0x7f74380b0d10 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.164+0000 7f742ea00640 1 -- 172.21.15.132:0/4265245499 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f74280598a0 con 0x7f74380b0d10 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.164+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 msgr2=0x7f457806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.164+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 0x7f457806ea40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4584036000 tx=0x7f458403c040 comp rx=0 tx=0).stop 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.164+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 msgr2=0x7f4598159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.164+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f458c00b4d0 tx=0x7f458c00b990 comp rx=0 tx=0).stop 2024-03-27T06:14:13.156 INFO:teuthology.orchestra.run.smithi132.stdout:55834574855 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 shutdown_connections 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45981572b0 0x7f45981f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f45980c64d0 0x7f45981f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f457806c580 0x7f457806ea40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 --2- 172.21.15.132:0/3801605615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4598070de0 0x7f4598159b30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 >> 172.21.15.132:0/3801605615 conn(0x7f45980cc7a0 msgr2=0x7f4598156400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 shutdown_connections 2024-03-27T06:14:13.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.166+0000 7f459da00640 1 -- 172.21.15.132:0/3801605615 wait complete. 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.168+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 msgr2=0x7f741806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.168+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 0x7f741806ea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74380c8810 tx=0x7f742403c040 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.168+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 msgr2=0x7f74380cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.168+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 0x7f74380cad50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f742800d550 tx=0x7f742800da10 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 shutdown_connections 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7438159c30 0x7f7438145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7438152c60 0x7f74381566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f741806c580 0x7f741806ea40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 --2- 172.21.15.132:0/4265245499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380b0d10 0x7f74380cad50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 >> 172.21.15.132:0/4265245499 conn(0x7f7438067280 msgr2=0x7f74380ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 shutdown_connections 2024-03-27T06:14:13.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.169+0000 7f743d800640 1 -- 172.21.15.132:0/4265245499 wait complete. 2024-03-27T06:14:13.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.170+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd75c005850 con 0x7fd778037180 2024-03-27T06:14:13.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.172+0000 7fd773e00640 1 -- 172.21.15.132:0/1313719550 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd77405c730 con 0x7fd778037180 2024-03-27T06:14:13.165 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-03-27T06:14:13.165 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.176+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 msgr2=0x7fd76006ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.176+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 0x7fd76006ebf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd76c036000 tx=0x7fd76c008030 comp rx=0 tx=0).stop 2024-03-27T06:14:13.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.176+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 msgr2=0x7fd77804ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.176+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c3f0 tx=0x7fd774001e70 comp rx=0 tx=0).stop 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 shutdown_connections 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7780d92c0 0x7fd778052bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd76006c730 0x7fd76006ebf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd778038d60 0x7fd7780d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 --2- 172.21.15.132:0/1313719550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd778037180 0x7fd77804ce40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 >> 172.21.15.132:0/1313719550 conn(0x7fd778054610 msgr2=0x7fd77804f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 shutdown_connections 2024-03-27T06:14:13.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.177+0000 7fd785a00640 1 -- 172.21.15.132:0/1313719550 wait complete. 2024-03-27T06:14:13.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.178+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fad2c068840 con 0x7fad10004800 2024-03-27T06:14:13.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.179+0000 7fad23400640 1 -- 172.21.15.132:0/3445426554 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fad1405d5b0 con 0x7fad10004800 2024-03-27T06:14:13.170 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.179+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9b64068840 con 0x7f9b6406b520 2024-03-27T06:14:13.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.180+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f9cbc0c7f60 con 0x7f9cbc15e320 2024-03-27T06:14:13.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.180+0000 7f9b60c00640 1 -- 172.21.15.132:0/835814704 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9b50061b30 con 0x7f9b6406b520 2024-03-27T06:14:13.171 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.181+0000 7f9cb6000640 1 -- 172.21.15.132:0/3434815371 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9cac07bea0 con 0x7f9cbc15e320 2024-03-27T06:14:13.173 INFO:teuthology.orchestra.run.smithi132.stdout:55834574855 2024-03-27T06:14:13.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.183+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 msgr2=0x7fad0806ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.183+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 0x7fad0806ec60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0d1160 tx=0x7fad1c008030 comp rx=0 tx=0).stop 2024-03-27T06:14:13.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.183+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 msgr2=0x7fad2c0b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.183+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad140017e0 tx=0x7fad140077a0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 shutdown_connections 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad0806c7a0 0x7fad0806ec60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad2c15d5b0 0x7fad2c156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad2c0b5fa0 0x7fad2c15d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 --2- 172.21.15.132:0/3445426554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad10004800 0x7fad2c0b5a60 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad140017e0 tx=0x7fad140077a0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 >> 172.21.15.132:0/3445426554 conn(0x7fad2c06dbd0 msgr2=0x7fad2c154870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 shutdown_connections 2024-03-27T06:14:13.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7fad32800640 1 -- 172.21.15.132:0/3445426554 wait complete. 2024-03-27T06:14:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 msgr2=0x7f9b4406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 0x7f9b4406e9f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b641512f0 tx=0x7f9b58009040 comp rx=0 tx=0).stop 2024-03-27T06:14:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 msgr2=0x7f9b6415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.184+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b50001b40 tx=0x7f9b50014420 comp rx=0 tx=0).stop 2024-03-27T06:14:13.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 shutdown_connections 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9b4406c530 0x7f9b4406e9f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b640c8160 0x7f9b641470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b6406b940 0x7f9b64146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 --2- 172.21.15.132:0/835814704 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9b6406b520 0x7f9b6415a420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 >> 172.21.15.132:0/835814704 conn(0x7f9b640cc060 msgr2=0x7f9b6414f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 shutdown_connections 2024-03-27T06:14:13.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9b6b600640 1 -- 172.21.15.132:0/835814704 wait complete. 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 msgr2=0x7f9ca006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 0x7f9ca006ea20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009b60 tx=0x7f9cb0008040 comp rx=0 tx=0).stop 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 msgr2=0x7f9cbc15cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15cc30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0708f0 tx=0x7f9cac0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 shutdown_connections 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cbc15e320 0x7f9cbc15cc30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9ca006c560 0x7f9ca006ea20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cbc15df40 0x7f9cbc200d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 --2- 172.21.15.132:0/3434815371 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9cbc0c12e0 0x7f9cbc06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.185+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 >> 172.21.15.132:0/3434815371 conn(0x7f9cbc06e790 msgr2=0x7f9cbc0ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.186+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 shutdown_connections 2024-03-27T06:14:13.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.186+0000 7f9cc3200640 1 -- 172.21.15.132:0/3434815371 wait complete. 2024-03-27T06:14:13.184 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-27T06:14:13.184 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-03-27T06:14:13.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.196+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0214154280 con 0x7f02140c8a30 2024-03-27T06:14:13.188 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.197+0000 7f020b400640 1 -- 172.21.15.132:0/2752829644 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f020c07c3b0 con 0x7f02140c8a30 2024-03-27T06:14:13.188 INFO:teuthology.orchestra.run.smithi132.stdout:55834574854 2024-03-27T06:14:13.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-03-27T06:14:13.192 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.201+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 msgr2=0x7f01f406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.201+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 0x7f01f406eac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0200000f60 tx=0x7f0200007a80 comp rx=0 tx=0).stop 2024-03-27T06:14:13.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.201+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 msgr2=0x7f021414c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.201+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 0x7f021414c400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f021406d6b0 tx=0x7f020c004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:13.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 shutdown_connections 2024-03-27T06:14:13.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f02140c8a30 0x7f021414c400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f01f406c600 0x7f01f406eac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021406cc80 0x7f021414bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 --2- 172.21.15.132:0/2752829644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f021406c8a0 0x7f0214153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 >> 172.21.15.132:0/2752829644 conn(0x7f02140ccbe0 msgr2=0x7f0214159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.202+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 shutdown_connections 2024-03-27T06:14:13.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.203+0000 7f021a800640 1 -- 172.21.15.132:0/2752829644 wait complete. 2024-03-27T06:14:13.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-27T06:14:13.200 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-27T06:14:13.200 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-03-27T06:14:13.202 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-03-27T06:14:13.212 DEBUG:teuthology.parallel:result is None 2024-03-27T06:14:13.212 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T06:14:13.213 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:13.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.466+0000 7f3b86a00640 1 Processor -- start 2024-03-27T06:14:13.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.466+0000 7f3b86a00640 1 -- start start 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.467+0000 7f3b86a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 0x7f3b800b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.467+0000 7f3b86a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 0x7f3b80073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.467+0000 7f3b86a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b800c80f0 con 0x7f3b80152740 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.467+0000 7f3b86a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8006cc40 con 0x7f3b800c5640 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.467+0000 7f3b86a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3b800c5cb0 con 0x7f3b80072ff0 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b86000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 0x7f3b80073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b84c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 0x7f3b800b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b85600640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3b80152740 0x7f3b8006c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37798/0 (socket says 172.21.15.132:37798) 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b85600640 1 -- 172.21.15.132:0/191679196 learned_addr learned my addr 172.21.15.132:0/191679196 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b7fe00640 1 -- 172.21.15.132:0/191679196 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3045817875 0 0) 0x7f3b800c80f0 con 0x7f3b80152740 2024-03-27T06:14:13.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b7fe00640 1 -- 172.21.15.132:0/191679196 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b68003520 con 0x7f3b80152740 2024-03-27T06:14:13.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b86000640 1 -- 172.21.15.132:0/191679196 >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 msgr2=0x7f3b800b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:13.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b86000640 1 --2- 172.21.15.132:0/191679196 >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 0x7f3b800b1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b86000640 1 -- 172.21.15.132:0/191679196 >> v1:172.21.15.134:6789/0 conn(0x7f3b80152740 legacy=0x7f3b8006c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b86000640 1 -- 172.21.15.132:0/191679196 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b800c5f90 con 0x7f3b80072ff0 2024-03-27T06:14:13.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.468+0000 7f3b84c00640 1 --2- 172.21.15.132:0/191679196 >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 0x7f3b800b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.469+0000 7f3b86000640 1 --2- 172.21.15.132:0/191679196 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 0x7f3b80073460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009d90 tx=0x7f3b7802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01dc56b84a4cb99 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.469+0000 7f3b7fe00640 1 -- 172.21.15.132:0/191679196 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78039070 con 0x7f3b80072ff0 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.469+0000 7f3b7fe00640 1 -- 172.21.15.132:0/191679196 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7802cce0 con 0x7f3b80072ff0 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.469+0000 7f3b7fe00640 1 -- 172.21.15.132:0/191679196 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78034660 con 0x7f3b80072ff0 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 -- 172.21.15.132:0/191679196 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 msgr2=0x7f3b80073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 --2- 172.21.15.132:0/191679196 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 0x7f3b80073460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009d90 tx=0x7f3b7802bf30 comp rx=0 tx=0).stop 2024-03-27T06:14:13.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 -- 172.21.15.132:0/191679196 shutdown_connections 2024-03-27T06:14:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 --2- 172.21.15.132:0/191679196 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b80072ff0 0x7f3b80073460 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 --2- 172.21.15.132:0/191679196 >> v2:172.21.15.134:3300/0 conn(0x7f3b800c5640 0x7f3b800b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 -- 172.21.15.132:0/191679196 >> 172.21.15.132:0/191679196 conn(0x7f3b800cc7a0 msgr2=0x7f3b800c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 -- 172.21.15.132:0/191679196 shutdown_connections 2024-03-27T06:14:13.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.470+0000 7f3b86a00640 1 -- 172.21.15.132:0/191679196 wait complete. 2024-03-27T06:14:13.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.472+0000 7f3b86a00640 1 Processor -- start 2024-03-27T06:14:13.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.472+0000 7f3b86a00640 1 -- start start 2024-03-27T06:14:13.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b800c5640 0x7f3b80149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b80152740 0x7f3b8014a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b80158db0 con 0x7f3b80072ff0 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3b80158ad0 con 0x7f3b80152740 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b86a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3b80152b20 con 0x7f3b800c5640 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.473+0000 7f3b85600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37478/0 (socket says 172.21.15.132:37478) 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 -- 172.21.15.132:0/1827974647 learned_addr learned my addr 172.21.15.132:0/1827974647 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b84c00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b800c5640 0x7f3b80149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b86000640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b80152740 0x7f3b8014a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 -- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b800c5640 msgr2=0x7f3b80149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b800c5640 0x7f3b80149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 -- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b80152740 msgr2=0x7f3b8014a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b80152740 0x7f3b8014a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 -- 172.21.15.132:0/1827974647 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b80159df0 con 0x7f3b80072ff0 2024-03-27T06:14:13.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b85600640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c3c0 tx=0x7f3b7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74017070 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b80159a10 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b74005ce0 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.474+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740072f0 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.475+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b801f3910 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.476+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b800686c0 con 0x7f3b80072ff0 2024-03-27T06:14:13.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.477+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3b7401a070 con 0x7f3b80072ff0 2024-03-27T06:14:13.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.477+0000 7f3b7d600640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 0x7f3b6006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.478+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3b7408a9c0 con 0x7f3b80072ff0 2024-03-27T06:14:13.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.478+0000 7f3b84c00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 0x7f3b6006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.478+0000 7f3b84c00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 0x7f3b6006ea20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009560 tx=0x7f3b6c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.482+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3b7408ce20 con 0x7f3b80072ff0 2024-03-27T06:14:13.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.674+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b8014c990 con 0x7f3b6006c560 2024-03-27T06:14:13.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.681+0000 7f3b7d600640 1 -- 172.21.15.132:0/1827974647 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251908 (secure 0 0 0) 0x7f3b8014c990 con 0x7f3b6006c560 2024-03-27T06:14:13.673 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:13.676 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.686+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 msgr2=0x7f3b6006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.686+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 0x7f3b6006ea20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009560 tx=0x7f3b6c009190 comp rx=0 tx=0).stop 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 msgr2=0x7f3b800c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c3c0 tx=0x7f3b7400c880 comp rx=0 tx=0).stop 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 shutdown_connections 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3b6006c560 0x7f3b6006ea20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b80152740 0x7f3b8014a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3b800c5640 0x7f3b80149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 --2- 172.21.15.132:0/1827974647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b80072ff0 0x7f3b800c3c30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 >> 172.21.15.132:0/1827974647 conn(0x7f3b800cc7a0 msgr2=0x7f3b80157ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 shutdown_connections 2024-03-27T06:14:13.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.687+0000 7f3b86a00640 1 -- 172.21.15.132:0/1827974647 wait complete. 2024-03-27T06:14:13.696 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-27T06:14:12.393054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218020,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756700,"statfs":{"total":773094113280,"available":772870860800,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.885252"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048058+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:07.048058+0000","last_peered":"2024-03-27T06:14:07.048058+0000","last_clean":"2024-03-27T06:14:07.048058+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:07.048058+0000","last_undegraded":"2024-03-27T06:14:07.048058+0000","last_fullsized":"2024-03-27T06:14:07.048058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708723+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:06.708723+0000","last_peered":"2024-03-27T06:14:06.708723+0000","last_clean":"2024-03-27T06:14:06.708723+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:06.708723+0000","last_undegraded":"2024-03-27T06:14:06.708723+0000","last_fullsized":"2024-03-27T06:14:06.708723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708489+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:06.708489+0000","last_peered":"2024-03-27T06:14:06.708489+0000","last_clean":"2024-03-27T06:14:06.708489+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:06.708489+0000","last_undegraded":"2024-03-27T06:14:06.708489+0000","last_fullsized":"2024-03-27T06:14:06.708489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462134+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:07.462134+0000","last_peered":"2024-03-27T06:14:07.462134+0000","last_clean":"2024-03-27T06:14:07.462134+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:07.462134+0000","last_undegraded":"2024-03-27T06:14:07.462134+0000","last_fullsized":"2024-03-27T06:14:07.462134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048224+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:07.048224+0000","last_peered":"2024-03-27T06:14:07.048224+0000","last_clean":"2024-03-27T06:14:07.048224+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:07.048224+0000","last_undegraded":"2024-03-27T06:14:07.048224+0000","last_fullsized":"2024-03-27T06:14:07.048224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462236+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:07.462236+0000","last_peered":"2024-03-27T06:14:07.462236+0000","last_clean":"2024-03-27T06:14:07.462236+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:07.462236+0000","last_undegraded":"2024-03-27T06:14:07.462236+0000","last_fullsized":"2024-03-27T06:14:07.462236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047358+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:07.047358+0000","last_peered":"2024-03-27T06:14:07.047358+0000","last_clean":"2024-03-27T06:14:07.047358+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:07.047358+0000","last_undegraded":"2024-03-27T06:14:07.047358+0000","last_fullsized":"2024-03-27T06:14:07.047358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708492+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:06.708492+0000","last_peered":"2024-03-27T06:14:06.708492+0000","last_clean":"2024-03-27T06:14:06.708492+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:06.708492+0000","last_undegraded":"2024-03-27T06:14:06.708492+0000","last_fullsized":"2024-03-27T06:14:06.708492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802323+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:06.802323+0000","last_peered":"2024-03-27T06:14:06.802323+0000","last_clean":"2024-03-27T06:14:06.802323+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:06.802323+0000","last_undegraded":"2024-03-27T06:14:06.802323+0000","last_fullsized":"2024-03-27T06:14:06.802323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048760+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:07.048760+0000","last_peered":"2024-03-27T06:14:07.048760+0000","last_clean":"2024-03-27T06:14:07.048760+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:07.048760+0000","last_undegraded":"2024-03-27T06:14:07.048760+0000","last_fullsized":"2024-03-27T06:14:07.048760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048755+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:07.048755+0000","last_peered":"2024-03-27T06:14:07.048755+0000","last_clean":"2024-03-27T06:14:07.048755+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:07.048755+0000","last_undegraded":"2024-03-27T06:14:07.048755+0000","last_fullsized":"2024-03-27T06:14:07.048755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047765+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:07.047765+0000","last_peered":"2024-03-27T06:14:07.047765+0000","last_clean":"2024-03-27T06:14:07.047765+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:07.047765+0000","last_undegraded":"2024-03-27T06:14:07.047765+0000","last_fullsized":"2024-03-27T06:14:07.047765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808956+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:07.808956+0000","last_peered":"2024-03-27T06:14:07.808956+0000","last_clean":"2024-03-27T06:14:07.808956+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:07.808956+0000","last_undegraded":"2024-03-27T06:14:07.808956+0000","last_fullsized":"2024-03-27T06:14:07.808956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808841+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:07.808841+0000","last_peered":"2024-03-27T06:14:07.808841+0000","last_clean":"2024-03-27T06:14:07.808841+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:07.808841+0000","last_undegraded":"2024-03-27T06:14:07.808841+0000","last_fullsized":"2024-03-27T06:14:07.808841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708767+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:06.708767+0000","last_peered":"2024-03-27T06:14:06.708767+0000","last_clean":"2024-03-27T06:14:06.708767+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:06.708767+0000","last_undegraded":"2024-03-27T06:14:06.708767+0000","last_fullsized":"2024-03-27T06:14:06.708767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048108+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:07.048108+0000","last_peered":"2024-03-27T06:14:07.048108+0000","last_clean":"2024-03-27T06:14:07.048108+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:07.048108+0000","last_undegraded":"2024-03-27T06:14:07.048108+0000","last_fullsized":"2024-03-27T06:14:07.048108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046295+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:07.046295+0000","last_peered":"2024-03-27T06:14:07.046295+0000","last_clean":"2024-03-27T06:14:07.046295+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:07.046295+0000","last_undegraded":"2024-03-27T06:14:07.046295+0000","last_fullsized":"2024-03-27T06:14:07.046295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801932+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:06.801932+0000","last_peered":"2024-03-27T06:14:06.801932+0000","last_clean":"2024-03-27T06:14:06.801932+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:06.801932+0000","last_undegraded":"2024-03-27T06:14:06.801932+0000","last_fullsized":"2024-03-27T06:14:06.801932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708558+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:06.708558+0000","last_peered":"2024-03-27T06:14:06.708558+0000","last_clean":"2024-03-27T06:14:06.708558+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:06.708558+0000","last_undegraded":"2024-03-27T06:14:06.708558+0000","last_fullsized":"2024-03-27T06:14:06.708558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708736+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:06.708736+0000","last_peered":"2024-03-27T06:14:06.708736+0000","last_clean":"2024-03-27T06:14:06.708736+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:06.708736+0000","last_undegraded":"2024-03-27T06:14:06.708736+0000","last_fullsized":"2024-03-27T06:14:06.708736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462807+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:07.462807+0000","last_peered":"2024-03-27T06:14:07.462807+0000","last_clean":"2024-03-27T06:14:07.462807+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:07.462807+0000","last_undegraded":"2024-03-27T06:14:07.462807+0000","last_fullsized":"2024-03-27T06:14:07.462807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809252+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:07.809252+0000","last_peered":"2024-03-27T06:14:07.809252+0000","last_clean":"2024-03-27T06:14:07.809252+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:07.809252+0000","last_undegraded":"2024-03-27T06:14:07.809252+0000","last_fullsized":"2024-03-27T06:14:07.809252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809134+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:07.809134+0000","last_peered":"2024-03-27T06:14:07.809134+0000","last_clean":"2024-03-27T06:14:07.809134+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:07.809134+0000","last_undegraded":"2024-03-27T06:14:07.809134+0000","last_fullsized":"2024-03-27T06:14:07.809134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708534+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:06.708534+0000","last_peered":"2024-03-27T06:14:06.708534+0000","last_clean":"2024-03-27T06:14:06.708534+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:06.708534+0000","last_undegraded":"2024-03-27T06:14:06.708534+0000","last_fullsized":"2024-03-27T06:14:06.708534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.461979+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:07.461979+0000","last_peered":"2024-03-27T06:14:07.461979+0000","last_clean":"2024-03-27T06:14:07.461979+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:07.461979+0000","last_undegraded":"2024-03-27T06:14:07.461979+0000","last_fullsized":"2024-03-27T06:14:07.461979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802104+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:06.802104+0000","last_peered":"2024-03-27T06:14:06.802104+0000","last_clean":"2024-03-27T06:14:06.802104+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:06.802104+0000","last_undegraded":"2024-03-27T06:14:06.802104+0000","last_fullsized":"2024-03-27T06:14:06.802104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708648+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:06.708648+0000","last_peered":"2024-03-27T06:14:06.708648+0000","last_clean":"2024-03-27T06:14:06.708648+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:06.708648+0000","last_undegraded":"2024-03-27T06:14:06.708648+0000","last_fullsized":"2024-03-27T06:14:06.708648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046346+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:07.046346+0000","last_peered":"2024-03-27T06:14:07.046346+0000","last_clean":"2024-03-27T06:14:07.046346+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:07.046346+0000","last_undegraded":"2024-03-27T06:14:07.046346+0000","last_fullsized":"2024-03-27T06:14:07.046346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431388+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:05.431388+0000","last_peered":"2024-03-27T06:14:05.431388+0000","last_clean":"2024-03-27T06:14:05.431388+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:05.431388+0000","last_undegraded":"2024-03-27T06:14:05.431388+0000","last_fullsized":"2024-03-27T06:14:05.431388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808715+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:07.808715+0000","last_peered":"2024-03-27T06:14:07.808715+0000","last_clean":"2024-03-27T06:14:07.808715+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:07.808715+0000","last_undegraded":"2024-03-27T06:14:07.808715+0000","last_fullsized":"2024-03-27T06:14:07.808715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708759+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:06.708759+0000","last_peered":"2024-03-27T06:14:06.708759+0000","last_clean":"2024-03-27T06:14:06.708759+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:06.708759+0000","last_undegraded":"2024-03-27T06:14:06.708759+0000","last_fullsized":"2024-03-27T06:14:06.708759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529878+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:04.529878+0000","last_peered":"2024-03-27T06:14:04.529878+0000","last_clean":"2024-03-27T06:14:04.529878+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:04.529878+0000","last_undegraded":"2024-03-27T06:14:04.529878+0000","last_fullsized":"2024-03-27T06:14:04.529878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046377+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:07.046377+0000","last_peered":"2024-03-27T06:14:07.046377+0000","last_clean":"2024-03-27T06:14:07.046377+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:07.046377+0000","last_undegraded":"2024-03-27T06:14:07.046377+0000","last_fullsized":"2024-03-27T06:14:07.046377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802296+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:06.802296+0000","last_peered":"2024-03-27T06:14:06.802296+0000","last_clean":"2024-03-27T06:14:06.802296+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:06.802296+0000","last_undegraded":"2024-03-27T06:14:06.802296+0000","last_fullsized":"2024-03-27T06:14:06.802296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462989+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:07.462989+0000","last_peered":"2024-03-27T06:14:07.462989+0000","last_clean":"2024-03-27T06:14:07.462989+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:07.462989+0000","last_undegraded":"2024-03-27T06:14:07.462989+0000","last_fullsized":"2024-03-27T06:14:07.462989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048943+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:07.048943+0000","last_peered":"2024-03-27T06:14:07.048943+0000","last_clean":"2024-03-27T06:14:07.048943+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:07.048943+0000","last_undegraded":"2024-03-27T06:14:07.048943+0000","last_fullsized":"2024-03-27T06:14:07.048943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708885+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:06.708885+0000","last_peered":"2024-03-27T06:14:06.708885+0000","last_clean":"2024-03-27T06:14:06.708885+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:06.708885+0000","last_undegraded":"2024-03-27T06:14:06.708885+0000","last_fullsized":"2024-03-27T06:14:06.708885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.515561+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:08.515561+0000","last_peered":"2024-03-27T06:14:08.515561+0000","last_clean":"2024-03-27T06:14:08.515561+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:08.515561+0000","last_undegraded":"2024-03-27T06:14:08.515561+0000","last_fullsized":"2024-03-27T06:14:08.515561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.529230+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:08.529230+0000","last_peered":"2024-03-27T06:14:08.529230+0000","last_clean":"2024-03-27T06:14:08.529230+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:08.529230+0000","last_undegraded":"2024-03-27T06:14:08.529230+0000","last_fullsized":"2024-03-27T06:14:08.529230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.049066+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:07.049066+0000","last_peered":"2024-03-27T06:14:07.049066+0000","last_clean":"2024-03-27T06:14:07.049066+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:07.049066+0000","last_undegraded":"2024-03-27T06:14:07.049066+0000","last_fullsized":"2024-03-27T06:14:07.049066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.521061+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:08.521061+0000","last_peered":"2024-03-27T06:14:08.521061+0000","last_clean":"2024-03-27T06:14:08.521061+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:08.521061+0000","last_undegraded":"2024-03-27T06:14:08.521061+0000","last_fullsized":"2024-03-27T06:14:08.521061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808902+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:07.808902+0000","last_peered":"2024-03-27T06:14:07.808902+0000","last_clean":"2024-03-27T06:14:07.808902+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:07.808902+0000","last_undegraded":"2024-03-27T06:14:07.808902+0000","last_fullsized":"2024-03-27T06:14:07.808902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530841+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:04.530841+0000","last_peered":"2024-03-27T06:14:04.530841+0000","last_clean":"2024-03-27T06:14:04.530841+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:04.530841+0000","last_undegraded":"2024-03-27T06:14:04.530841+0000","last_fullsized":"2024-03-27T06:14:04.530841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047314+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:07.047314+0000","last_peered":"2024-03-27T06:14:07.047314+0000","last_clean":"2024-03-27T06:14:07.047314+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:07.047314+0000","last_undegraded":"2024-03-27T06:14:07.047314+0000","last_fullsized":"2024-03-27T06:14:07.047314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802187+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:06.802187+0000","last_peered":"2024-03-27T06:14:06.802187+0000","last_clean":"2024-03-27T06:14:06.802187+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:06.802187+0000","last_undegraded":"2024-03-27T06:14:06.802187+0000","last_fullsized":"2024-03-27T06:14:06.802187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.519003+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:08.519003+0000","last_peered":"2024-03-27T06:14:08.519003+0000","last_clean":"2024-03-27T06:14:08.519003+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:08.519003+0000","last_undegraded":"2024-03-27T06:14:08.519003+0000","last_fullsized":"2024-03-27T06:14:08.519003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048234+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:07.048234+0000","last_peered":"2024-03-27T06:14:07.048234+0000","last_clean":"2024-03-27T06:14:07.048234+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:07.048234+0000","last_undegraded":"2024-03-27T06:14:07.048234+0000","last_fullsized":"2024-03-27T06:14:07.048234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.463061+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:07.463061+0000","last_peered":"2024-03-27T06:14:07.463061+0000","last_clean":"2024-03-27T06:14:07.463061+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:07.463061+0000","last_undegraded":"2024-03-27T06:14:07.463061+0000","last_fullsized":"2024-03-27T06:14:07.463061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.517650+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:08.517650+0000","last_peered":"2024-03-27T06:14:08.517650+0000","last_clean":"2024-03-27T06:14:08.517650+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:08.517650+0000","last_undegraded":"2024-03-27T06:14:08.517650+0000","last_fullsized":"2024-03-27T06:14:08.517650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047986+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:07.047986+0000","last_peered":"2024-03-27T06:14:07.047986+0000","last_clean":"2024-03-27T06:14:07.047986+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:07.047986+0000","last_undegraded":"2024-03-27T06:14:07.047986+0000","last_fullsized":"2024-03-27T06:14:07.047986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708579+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:06.708579+0000","last_peered":"2024-03-27T06:14:06.708579+0000","last_clean":"2024-03-27T06:14:06.708579+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:06.708579+0000","last_undegraded":"2024-03-27T06:14:06.708579+0000","last_fullsized":"2024-03-27T06:14:06.708579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048156+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:07.048156+0000","last_peered":"2024-03-27T06:14:07.048156+0000","last_clean":"2024-03-27T06:14:07.048156+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:07.048156+0000","last_undegraded":"2024-03-27T06:14:07.048156+0000","last_fullsized":"2024-03-27T06:14:07.048156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809247+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:07.809247+0000","last_peered":"2024-03-27T06:14:07.809247+0000","last_clean":"2024-03-27T06:14:07.809247+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:07.809247+0000","last_undegraded":"2024-03-27T06:14:07.809247+0000","last_fullsized":"2024-03-27T06:14:07.809247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801767+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:06.801767+0000","last_peered":"2024-03-27T06:14:06.801767+0000","last_clean":"2024-03-27T06:14:06.801767+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:06.801767+0000","last_undegraded":"2024-03-27T06:14:06.801767+0000","last_fullsized":"2024-03-27T06:14:06.801767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531520+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:04.531520+0000","last_peered":"2024-03-27T06:14:04.531520+0000","last_clean":"2024-03-27T06:14:04.531520+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:04.531520+0000","last_undegraded":"2024-03-27T06:14:04.531520+0000","last_fullsized":"2024-03-27T06:14:04.531520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529895+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:04.529895+0000","last_peered":"2024-03-27T06:14:04.529895+0000","last_clean":"2024-03-27T06:14:04.529895+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:04.529895+0000","last_undegraded":"2024-03-27T06:14:04.529895+0000","last_fullsized":"2024-03-27T06:14:04.529895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808793+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:07.808793+0000","last_peered":"2024-03-27T06:14:07.808793+0000","last_clean":"2024-03-27T06:14:07.808793+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:07.808793+0000","last_undegraded":"2024-03-27T06:14:07.808793+0000","last_fullsized":"2024-03-27T06:14:07.808793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527370+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:04.527370+0000","last_peered":"2024-03-27T06:14:04.527370+0000","last_clean":"2024-03-27T06:14:04.527370+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:04.527370+0000","last_undegraded":"2024-03-27T06:14:04.527370+0000","last_fullsized":"2024-03-27T06:14:04.527370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708423+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:06.708423+0000","last_peered":"2024-03-27T06:14:06.708423+0000","last_clean":"2024-03-27T06:14:06.708423+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:06.708423+0000","last_undegraded":"2024-03-27T06:14:06.708423+0000","last_fullsized":"2024-03-27T06:14:06.708423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531072+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:04.531072+0000","last_peered":"2024-03-27T06:14:04.531072+0000","last_clean":"2024-03-27T06:14:04.531072+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:04.531072+0000","last_undegraded":"2024-03-27T06:14:04.531072+0000","last_fullsized":"2024-03-27T06:14:04.531072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048444+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:07.048444+0000","last_peered":"2024-03-27T06:14:07.048444+0000","last_clean":"2024-03-27T06:14:07.048444+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:07.048444+0000","last_undegraded":"2024-03-27T06:14:07.048444+0000","last_fullsized":"2024-03-27T06:14:07.048444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708415+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:06.708415+0000","last_peered":"2024-03-27T06:14:06.708415+0000","last_clean":"2024-03-27T06:14:06.708415+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:06.708415+0000","last_undegraded":"2024-03-27T06:14:06.708415+0000","last_fullsized":"2024-03-27T06:14:06.708415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462059+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:07.462059+0000","last_peered":"2024-03-27T06:14:07.462059+0000","last_clean":"2024-03-27T06:14:07.462059+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:07.462059+0000","last_undegraded":"2024-03-27T06:14:07.462059+0000","last_fullsized":"2024-03-27T06:14:07.462059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530500+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:04.530500+0000","last_peered":"2024-03-27T06:14:04.530500+0000","last_clean":"2024-03-27T06:14:04.530500+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:04.530500+0000","last_undegraded":"2024-03-27T06:14:04.530500+0000","last_fullsized":"2024-03-27T06:14:04.530500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809116+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:07.809116+0000","last_peered":"2024-03-27T06:14:07.809116+0000","last_clean":"2024-03-27T06:14:07.809116+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:07.809116+0000","last_undegraded":"2024-03-27T06:14:07.809116+0000","last_fullsized":"2024-03-27T06:14:07.809116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046678+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:07.046678+0000","last_peered":"2024-03-27T06:14:07.046678+0000","last_clean":"2024-03-27T06:14:07.046678+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:07.046678+0000","last_undegraded":"2024-03-27T06:14:07.046678+0000","last_fullsized":"2024-03-27T06:14:07.046678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046680+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:07.046680+0000","last_peered":"2024-03-27T06:14:07.046680+0000","last_clean":"2024-03-27T06:14:07.046680+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:07.046680+0000","last_undegraded":"2024-03-27T06:14:07.046680+0000","last_fullsized":"2024-03-27T06:14:07.046680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.527343+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:08.527343+0000","last_peered":"2024-03-27T06:14:08.527343+0000","last_clean":"2024-03-27T06:14:08.527343+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:08.527343+0000","last_undegraded":"2024-03-27T06:14:08.527343+0000","last_fullsized":"2024-03-27T06:14:08.527343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046583+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:07.046583+0000","last_peered":"2024-03-27T06:14:07.046583+0000","last_clean":"2024-03-27T06:14:07.046583+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:07.046583+0000","last_undegraded":"2024-03-27T06:14:07.046583+0000","last_fullsized":"2024-03-27T06:14:07.046583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046901+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:07.046901+0000","last_peered":"2024-03-27T06:14:07.046901+0000","last_clean":"2024-03-27T06:14:07.046901+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:07.046901+0000","last_undegraded":"2024-03-27T06:14:07.046901+0000","last_fullsized":"2024-03-27T06:14:07.046901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046348+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:07.046348+0000","last_peered":"2024-03-27T06:14:07.046348+0000","last_clean":"2024-03-27T06:14:07.046348+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:07.046348+0000","last_undegraded":"2024-03-27T06:14:07.046348+0000","last_fullsized":"2024-03-27T06:14:07.046348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808963+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:07.808963+0000","last_peered":"2024-03-27T06:14:07.808963+0000","last_clean":"2024-03-27T06:14:07.808963+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:07.808963+0000","last_undegraded":"2024-03-27T06:14:07.808963+0000","last_fullsized":"2024-03-27T06:14:07.808963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.526612+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:08.526612+0000","last_peered":"2024-03-27T06:14:08.526612+0000","last_clean":"2024-03-27T06:14:08.526612+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:08.526612+0000","last_undegraded":"2024-03-27T06:14:08.526612+0000","last_fullsized":"2024-03-27T06:14:08.526612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525809+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:08.525809+0000","last_peered":"2024-03-27T06:14:08.525809+0000","last_clean":"2024-03-27T06:14:08.525809+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:08.525809+0000","last_undegraded":"2024-03-27T06:14:08.525809+0000","last_fullsized":"2024-03-27T06:14:08.525809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802159+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:06.802159+0000","last_peered":"2024-03-27T06:14:06.802159+0000","last_clean":"2024-03-27T06:14:06.802159+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:06.802159+0000","last_undegraded":"2024-03-27T06:14:06.802159+0000","last_fullsized":"2024-03-27T06:14:06.802159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525502+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:08.525502+0000","last_peered":"2024-03-27T06:14:08.525502+0000","last_clean":"2024-03-27T06:14:08.525502+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:08.525502+0000","last_undegraded":"2024-03-27T06:14:08.525502+0000","last_fullsized":"2024-03-27T06:14:08.525502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708697+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:06.708697+0000","last_peered":"2024-03-27T06:14:06.708697+0000","last_clean":"2024-03-27T06:14:06.708697+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:06.708697+0000","last_undegraded":"2024-03-27T06:14:06.708697+0000","last_fullsized":"2024-03-27T06:14:06.708697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708881+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:06.708881+0000","last_peered":"2024-03-27T06:14:06.708881+0000","last_clean":"2024-03-27T06:14:06.708881+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:06.708881+0000","last_undegraded":"2024-03-27T06:14:06.708881+0000","last_fullsized":"2024-03-27T06:14:06.708881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708688+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:06.708688+0000","last_peered":"2024-03-27T06:14:06.708688+0000","last_clean":"2024-03-27T06:14:06.708688+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:06.708688+0000","last_undegraded":"2024-03-27T06:14:06.708688+0000","last_fullsized":"2024-03-27T06:14:06.708688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525412+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:08.525412+0000","last_peered":"2024-03-27T06:14:08.525412+0000","last_clean":"2024-03-27T06:14:08.525412+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:08.525412+0000","last_undegraded":"2024-03-27T06:14:08.525412+0000","last_fullsized":"2024-03-27T06:14:08.525412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666065,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666043,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:13.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:13.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.951+0000 7f4fc8000640 1 Processor -- start 2024-03-27T06:14:13.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.951+0000 7f4fc8000640 1 -- start start 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc8000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4fc0152950 0x7f4fc01590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc8000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 0x7f4fc0071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc8000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fc006b630 con 0x7f4fc0073970 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc8000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4fc0152d30 con 0x7f4fc0152950 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc8000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc006b8d0 con 0x7f4fc0073550 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4fc0152950 0x7f4fc01590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 0x7f4fc0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4fc0152950 0x7f4fc01590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37484/0 (socket says 172.21.15.132:37484) 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 0x7f4fc0071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35354/0 (socket says 172.21.15.132:35354) 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 -- 172.21.15.132:0/2970394727 learned_addr learned my addr 172.21.15.132:0/2970394727 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 -- 172.21.15.132:0/2970394727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 msgr2=0x7f4fc0071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 --2- 172.21.15.132:0/2970394727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 0x7f4fc0071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 -- 172.21.15.132:0/2970394727 >> v1:172.21.15.134:6789/0 conn(0x7f4fc0073970 legacy=0x7f4fc006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fc5600640 1 -- 172.21.15.132:0/2970394727 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc006bbb0 con 0x7f4fc0152950 2024-03-27T06:14:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.953+0000 7f4fbfe00640 1 -- 172.21.15.132:0/2970394727 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561024965 0 0) 0x7f4fc006b630 con 0x7f4fc0073970 2024-03-27T06:14:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.954+0000 7f4fc5600640 1 --2- 172.21.15.132:0/2970394727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc01590a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009870 tx=0x7f4fb402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc177b46631592b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fbfe00640 1 -- 172.21.15.132:0/2970394727 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4034070 con 0x7f4fc0152950 2024-03-27T06:14:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fbfe00640 1 -- 172.21.15.132:0/2970394727 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb4030840 con 0x7f4fc0152950 2024-03-27T06:14:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fbfe00640 1 -- 172.21.15.132:0/2970394727 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4030c10 con 0x7f4fc0152950 2024-03-27T06:14:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 -- 172.21.15.132:0/2970394727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 msgr2=0x7f4fc01590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2970394727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc01590a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009870 tx=0x7f4fb402b980 comp rx=0 tx=0).stop 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 -- 172.21.15.132:0/2970394727 shutdown_connections 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2970394727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073550 0x7f4fc0071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2970394727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc01590a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 -- 172.21.15.132:0/2970394727 >> 172.21.15.132:0/2970394727 conn(0x7f4fc00cc060 msgr2=0x7f4fc00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 -- 172.21.15.132:0/2970394727 shutdown_connections 2024-03-27T06:14:13.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.955+0000 7f4fc8000640 1 -- 172.21.15.132:0/2970394727 wait complete. 2024-03-27T06:14:13.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.957+0000 7f4fc8000640 1 Processor -- start 2024-03-27T06:14:13.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.957+0000 7f4fc8000640 1 -- start start 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 0x7f4fc01f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 0x7f4fc0156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc00c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc006c450 con 0x7f4fc0152950 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc006c170 con 0x7f4fc0073550 2024-03-27T06:14:13.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc8000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc006d660 con 0x7f4fc0073970 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc5600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 0x7f4fc01f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 0x7f4fc0156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc5600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 0x7f4fc01f05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56404/0 (socket says 172.21.15.132:56404) 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc5600640 1 -- 172.21.15.132:0/2497068761 learned_addr learned my addr 172.21.15.132:0/2497068761 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.958+0000 7f4fc6000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc00c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 -- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 msgr2=0x7f4fc01f05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 0x7f4fc01f05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 -- 172.21.15.132:0/2497068761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 msgr2=0x7f4fc00c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc00c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 -- 172.21.15.132:0/2497068761 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc00c4980 con 0x7f4fc0073970 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc4c00640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 0x7f4fc0156b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00e880 tx=0x7f4fac00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac017070 con 0x7f4fc0073970 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fac00c940 con 0x7f4fc0073970 2024-03-27T06:14:13.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00cd30 con 0x7f4fc0073970 2024-03-27T06:14:13.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc01566a0 con 0x7f4fc0073970 2024-03-27T06:14:13.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.959+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc00c40a0 con 0x7f4fc0073970 2024-03-27T06:14:13.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.960+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc0068840 con 0x7f4fc0073970 2024-03-27T06:14:13.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.961+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4fac01a070 con 0x7f4fc0073970 2024-03-27T06:14:13.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.962+0000 7f4fbd600640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 0x7f4fa006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:13.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.962+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4fac08b600 con 0x7f4fc0073970 2024-03-27T06:14:13.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.962+0000 7f4fc5600640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 0x7f4fa006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:13.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.963+0000 7f4fc5600640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 0x7f4fa006ea20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009870 tx=0x7f4fb403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:13.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:13.966+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4fac059530 con 0x7f4fc0073970 2024-03-27T06:14:14.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.147+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4fc00c6b30 con 0x7f4fa006c560 2024-03-27T06:14:14.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.154+0000 7f4fbd600640 1 -- 172.21.15.132:0/2497068761 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251908 (secure 0 0 0) 0x7f4fb4087820 con 0x7f4fa006c560 2024-03-27T06:14:14.148 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:14.149 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:14.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.158+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 msgr2=0x7f4fa006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.158+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 0x7f4fa006ea20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009870 tx=0x7f4fb403c040 comp rx=0 tx=0).stop 2024-03-27T06:14:14.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.158+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 msgr2=0x7f4fc0156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.158+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 0x7f4fc0156b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00e880 tx=0x7f4fac00ed40 comp rx=0 tx=0).stop 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 shutdown_connections 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fc0152950 0x7f4fc00c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4fc0073970 0x7f4fc0156b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4fa006c560 0x7f4fa006ea20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 --2- 172.21.15.132:0/2497068761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fc0073550 0x7f4fc01f05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 >> 172.21.15.132:0/2497068761 conn(0x7f4fc00cc060 msgr2=0x7f4fc00cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 shutdown_connections 2024-03-27T06:14:14.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.159+0000 7f4fc8000640 1 -- 172.21.15.132:0/2497068761 wait complete. 2024-03-27T06:14:14.167 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-27T06:14:12.393054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218020,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756700,"statfs":{"total":773094113280,"available":772870860800,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.885252"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048058+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:07.048058+0000","last_peered":"2024-03-27T06:14:07.048058+0000","last_clean":"2024-03-27T06:14:07.048058+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:07.048058+0000","last_undegraded":"2024-03-27T06:14:07.048058+0000","last_fullsized":"2024-03-27T06:14:07.048058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708723+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:06.708723+0000","last_peered":"2024-03-27T06:14:06.708723+0000","last_clean":"2024-03-27T06:14:06.708723+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:06.708723+0000","last_undegraded":"2024-03-27T06:14:06.708723+0000","last_fullsized":"2024-03-27T06:14:06.708723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708489+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:06.708489+0000","last_peered":"2024-03-27T06:14:06.708489+0000","last_clean":"2024-03-27T06:14:06.708489+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:06.708489+0000","last_undegraded":"2024-03-27T06:14:06.708489+0000","last_fullsized":"2024-03-27T06:14:06.708489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462134+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:07.462134+0000","last_peered":"2024-03-27T06:14:07.462134+0000","last_clean":"2024-03-27T06:14:07.462134+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:07.462134+0000","last_undegraded":"2024-03-27T06:14:07.462134+0000","last_fullsized":"2024-03-27T06:14:07.462134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048224+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:07.048224+0000","last_peered":"2024-03-27T06:14:07.048224+0000","last_clean":"2024-03-27T06:14:07.048224+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:07.048224+0000","last_undegraded":"2024-03-27T06:14:07.048224+0000","last_fullsized":"2024-03-27T06:14:07.048224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462236+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:07.462236+0000","last_peered":"2024-03-27T06:14:07.462236+0000","last_clean":"2024-03-27T06:14:07.462236+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:07.462236+0000","last_undegraded":"2024-03-27T06:14:07.462236+0000","last_fullsized":"2024-03-27T06:14:07.462236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047358+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:07.047358+0000","last_peered":"2024-03-27T06:14:07.047358+0000","last_clean":"2024-03-27T06:14:07.047358+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:07.047358+0000","last_undegraded":"2024-03-27T06:14:07.047358+0000","last_fullsized":"2024-03-27T06:14:07.047358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708492+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:06.708492+0000","last_peered":"2024-03-27T06:14:06.708492+0000","last_clean":"2024-03-27T06:14:06.708492+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:06.708492+0000","last_undegraded":"2024-03-27T06:14:06.708492+0000","last_fullsized":"2024-03-27T06:14:06.708492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802323+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:06.802323+0000","last_peered":"2024-03-27T06:14:06.802323+0000","last_clean":"2024-03-27T06:14:06.802323+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:06.802323+0000","last_undegraded":"2024-03-27T06:14:06.802323+0000","last_fullsized":"2024-03-27T06:14:06.802323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048760+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:07.048760+0000","last_peered":"2024-03-27T06:14:07.048760+0000","last_clean":"2024-03-27T06:14:07.048760+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:07.048760+0000","last_undegraded":"2024-03-27T06:14:07.048760+0000","last_fullsized":"2024-03-27T06:14:07.048760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048755+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:07.048755+0000","last_peered":"2024-03-27T06:14:07.048755+0000","last_clean":"2024-03-27T06:14:07.048755+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:07.048755+0000","last_undegraded":"2024-03-27T06:14:07.048755+0000","last_fullsized":"2024-03-27T06:14:07.048755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047765+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:07.047765+0000","last_peered":"2024-03-27T06:14:07.047765+0000","last_clean":"2024-03-27T06:14:07.047765+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:07.047765+0000","last_undegraded":"2024-03-27T06:14:07.047765+0000","last_fullsized":"2024-03-27T06:14:07.047765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808956+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:07.808956+0000","last_peered":"2024-03-27T06:14:07.808956+0000","last_clean":"2024-03-27T06:14:07.808956+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:07.808956+0000","last_undegraded":"2024-03-27T06:14:07.808956+0000","last_fullsized":"2024-03-27T06:14:07.808956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808841+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:07.808841+0000","last_peered":"2024-03-27T06:14:07.808841+0000","last_clean":"2024-03-27T06:14:07.808841+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:07.808841+0000","last_undegraded":"2024-03-27T06:14:07.808841+0000","last_fullsized":"2024-03-27T06:14:07.808841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708767+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:06.708767+0000","last_peered":"2024-03-27T06:14:06.708767+0000","last_clean":"2024-03-27T06:14:06.708767+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:06.708767+0000","last_undegraded":"2024-03-27T06:14:06.708767+0000","last_fullsized":"2024-03-27T06:14:06.708767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048108+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:07.048108+0000","last_peered":"2024-03-27T06:14:07.048108+0000","last_clean":"2024-03-27T06:14:07.048108+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:07.048108+0000","last_undegraded":"2024-03-27T06:14:07.048108+0000","last_fullsized":"2024-03-27T06:14:07.048108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046295+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:07.046295+0000","last_peered":"2024-03-27T06:14:07.046295+0000","last_clean":"2024-03-27T06:14:07.046295+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:07.046295+0000","last_undegraded":"2024-03-27T06:14:07.046295+0000","last_fullsized":"2024-03-27T06:14:07.046295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801932+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:06.801932+0000","last_peered":"2024-03-27T06:14:06.801932+0000","last_clean":"2024-03-27T06:14:06.801932+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:06.801932+0000","last_undegraded":"2024-03-27T06:14:06.801932+0000","last_fullsized":"2024-03-27T06:14:06.801932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708558+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:06.708558+0000","last_peered":"2024-03-27T06:14:06.708558+0000","last_clean":"2024-03-27T06:14:06.708558+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:06.708558+0000","last_undegraded":"2024-03-27T06:14:06.708558+0000","last_fullsized":"2024-03-27T06:14:06.708558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708736+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:06.708736+0000","last_peered":"2024-03-27T06:14:06.708736+0000","last_clean":"2024-03-27T06:14:06.708736+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:06.708736+0000","last_undegraded":"2024-03-27T06:14:06.708736+0000","last_fullsized":"2024-03-27T06:14:06.708736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462807+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:07.462807+0000","last_peered":"2024-03-27T06:14:07.462807+0000","last_clean":"2024-03-27T06:14:07.462807+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:07.462807+0000","last_undegraded":"2024-03-27T06:14:07.462807+0000","last_fullsized":"2024-03-27T06:14:07.462807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809252+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:07.809252+0000","last_peered":"2024-03-27T06:14:07.809252+0000","last_clean":"2024-03-27T06:14:07.809252+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:07.809252+0000","last_undegraded":"2024-03-27T06:14:07.809252+0000","last_fullsized":"2024-03-27T06:14:07.809252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809134+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:07.809134+0000","last_peered":"2024-03-27T06:14:07.809134+0000","last_clean":"2024-03-27T06:14:07.809134+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:07.809134+0000","last_undegraded":"2024-03-27T06:14:07.809134+0000","last_fullsized":"2024-03-27T06:14:07.809134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708534+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:06.708534+0000","last_peered":"2024-03-27T06:14:06.708534+0000","last_clean":"2024-03-27T06:14:06.708534+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:06.708534+0000","last_undegraded":"2024-03-27T06:14:06.708534+0000","last_fullsized":"2024-03-27T06:14:06.708534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.461979+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:07.461979+0000","last_peered":"2024-03-27T06:14:07.461979+0000","last_clean":"2024-03-27T06:14:07.461979+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:07.461979+0000","last_undegraded":"2024-03-27T06:14:07.461979+0000","last_fullsized":"2024-03-27T06:14:07.461979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802104+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:06.802104+0000","last_peered":"2024-03-27T06:14:06.802104+0000","last_clean":"2024-03-27T06:14:06.802104+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:06.802104+0000","last_undegraded":"2024-03-27T06:14:06.802104+0000","last_fullsized":"2024-03-27T06:14:06.802104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708648+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:06.708648+0000","last_peered":"2024-03-27T06:14:06.708648+0000","last_clean":"2024-03-27T06:14:06.708648+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:06.708648+0000","last_undegraded":"2024-03-27T06:14:06.708648+0000","last_fullsized":"2024-03-27T06:14:06.708648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046346+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:07.046346+0000","last_peered":"2024-03-27T06:14:07.046346+0000","last_clean":"2024-03-27T06:14:07.046346+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:07.046346+0000","last_undegraded":"2024-03-27T06:14:07.046346+0000","last_fullsized":"2024-03-27T06:14:07.046346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431388+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:05.431388+0000","last_peered":"2024-03-27T06:14:05.431388+0000","last_clean":"2024-03-27T06:14:05.431388+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:05.431388+0000","last_undegraded":"2024-03-27T06:14:05.431388+0000","last_fullsized":"2024-03-27T06:14:05.431388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808715+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:07.808715+0000","last_peered":"2024-03-27T06:14:07.808715+0000","last_clean":"2024-03-27T06:14:07.808715+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:07.808715+0000","last_undegraded":"2024-03-27T06:14:07.808715+0000","last_fullsized":"2024-03-27T06:14:07.808715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708759+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:06.708759+0000","last_peered":"2024-03-27T06:14:06.708759+0000","last_clean":"2024-03-27T06:14:06.708759+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:06.708759+0000","last_undegraded":"2024-03-27T06:14:06.708759+0000","last_fullsized":"2024-03-27T06:14:06.708759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529878+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:04.529878+0000","last_peered":"2024-03-27T06:14:04.529878+0000","last_clean":"2024-03-27T06:14:04.529878+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:04.529878+0000","last_undegraded":"2024-03-27T06:14:04.529878+0000","last_fullsized":"2024-03-27T06:14:04.529878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046377+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:07.046377+0000","last_peered":"2024-03-27T06:14:07.046377+0000","last_clean":"2024-03-27T06:14:07.046377+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:07.046377+0000","last_undegraded":"2024-03-27T06:14:07.046377+0000","last_fullsized":"2024-03-27T06:14:07.046377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802296+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:06.802296+0000","last_peered":"2024-03-27T06:14:06.802296+0000","last_clean":"2024-03-27T06:14:06.802296+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:06.802296+0000","last_undegraded":"2024-03-27T06:14:06.802296+0000","last_fullsized":"2024-03-27T06:14:06.802296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462989+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:07.462989+0000","last_peered":"2024-03-27T06:14:07.462989+0000","last_clean":"2024-03-27T06:14:07.462989+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:07.462989+0000","last_undegraded":"2024-03-27T06:14:07.462989+0000","last_fullsized":"2024-03-27T06:14:07.462989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048943+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:07.048943+0000","last_peered":"2024-03-27T06:14:07.048943+0000","last_clean":"2024-03-27T06:14:07.048943+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:07.048943+0000","last_undegraded":"2024-03-27T06:14:07.048943+0000","last_fullsized":"2024-03-27T06:14:07.048943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708885+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:06.708885+0000","last_peered":"2024-03-27T06:14:06.708885+0000","last_clean":"2024-03-27T06:14:06.708885+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:06.708885+0000","last_undegraded":"2024-03-27T06:14:06.708885+0000","last_fullsized":"2024-03-27T06:14:06.708885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.515561+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:08.515561+0000","last_peered":"2024-03-27T06:14:08.515561+0000","last_clean":"2024-03-27T06:14:08.515561+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:08.515561+0000","last_undegraded":"2024-03-27T06:14:08.515561+0000","last_fullsized":"2024-03-27T06:14:08.515561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.529230+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:08.529230+0000","last_peered":"2024-03-27T06:14:08.529230+0000","last_clean":"2024-03-27T06:14:08.529230+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:08.529230+0000","last_undegraded":"2024-03-27T06:14:08.529230+0000","last_fullsized":"2024-03-27T06:14:08.529230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.049066+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:07.049066+0000","last_peered":"2024-03-27T06:14:07.049066+0000","last_clean":"2024-03-27T06:14:07.049066+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:07.049066+0000","last_undegraded":"2024-03-27T06:14:07.049066+0000","last_fullsized":"2024-03-27T06:14:07.049066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.521061+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:08.521061+0000","last_peered":"2024-03-27T06:14:08.521061+0000","last_clean":"2024-03-27T06:14:08.521061+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:08.521061+0000","last_undegraded":"2024-03-27T06:14:08.521061+0000","last_fullsized":"2024-03-27T06:14:08.521061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808902+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:07.808902+0000","last_peered":"2024-03-27T06:14:07.808902+0000","last_clean":"2024-03-27T06:14:07.808902+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:07.808902+0000","last_undegraded":"2024-03-27T06:14:07.808902+0000","last_fullsized":"2024-03-27T06:14:07.808902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530841+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:04.530841+0000","last_peered":"2024-03-27T06:14:04.530841+0000","last_clean":"2024-03-27T06:14:04.530841+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:04.530841+0000","last_undegraded":"2024-03-27T06:14:04.530841+0000","last_fullsized":"2024-03-27T06:14:04.530841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047314+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:07.047314+0000","last_peered":"2024-03-27T06:14:07.047314+0000","last_clean":"2024-03-27T06:14:07.047314+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:07.047314+0000","last_undegraded":"2024-03-27T06:14:07.047314+0000","last_fullsized":"2024-03-27T06:14:07.047314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802187+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:06.802187+0000","last_peered":"2024-03-27T06:14:06.802187+0000","last_clean":"2024-03-27T06:14:06.802187+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:06.802187+0000","last_undegraded":"2024-03-27T06:14:06.802187+0000","last_fullsized":"2024-03-27T06:14:06.802187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.519003+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:08.519003+0000","last_peered":"2024-03-27T06:14:08.519003+0000","last_clean":"2024-03-27T06:14:08.519003+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:08.519003+0000","last_undegraded":"2024-03-27T06:14:08.519003+0000","last_fullsized":"2024-03-27T06:14:08.519003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048234+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:07.048234+0000","last_peered":"2024-03-27T06:14:07.048234+0000","last_clean":"2024-03-27T06:14:07.048234+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:07.048234+0000","last_undegraded":"2024-03-27T06:14:07.048234+0000","last_fullsized":"2024-03-27T06:14:07.048234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.463061+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:07.463061+0000","last_peered":"2024-03-27T06:14:07.463061+0000","last_clean":"2024-03-27T06:14:07.463061+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:07.463061+0000","last_undegraded":"2024-03-27T06:14:07.463061+0000","last_fullsized":"2024-03-27T06:14:07.463061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.517650+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:08.517650+0000","last_peered":"2024-03-27T06:14:08.517650+0000","last_clean":"2024-03-27T06:14:08.517650+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:08.517650+0000","last_undegraded":"2024-03-27T06:14:08.517650+0000","last_fullsized":"2024-03-27T06:14:08.517650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047986+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:07.047986+0000","last_peered":"2024-03-27T06:14:07.047986+0000","last_clean":"2024-03-27T06:14:07.047986+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:07.047986+0000","last_undegraded":"2024-03-27T06:14:07.047986+0000","last_fullsized":"2024-03-27T06:14:07.047986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708579+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:06.708579+0000","last_peered":"2024-03-27T06:14:06.708579+0000","last_clean":"2024-03-27T06:14:06.708579+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:06.708579+0000","last_undegraded":"2024-03-27T06:14:06.708579+0000","last_fullsized":"2024-03-27T06:14:06.708579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048156+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:07.048156+0000","last_peered":"2024-03-27T06:14:07.048156+0000","last_clean":"2024-03-27T06:14:07.048156+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:07.048156+0000","last_undegraded":"2024-03-27T06:14:07.048156+0000","last_fullsized":"2024-03-27T06:14:07.048156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809247+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:07.809247+0000","last_peered":"2024-03-27T06:14:07.809247+0000","last_clean":"2024-03-27T06:14:07.809247+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:07.809247+0000","last_undegraded":"2024-03-27T06:14:07.809247+0000","last_fullsized":"2024-03-27T06:14:07.809247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801767+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:06.801767+0000","last_peered":"2024-03-27T06:14:06.801767+0000","last_clean":"2024-03-27T06:14:06.801767+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:06.801767+0000","last_undegraded":"2024-03-27T06:14:06.801767+0000","last_fullsized":"2024-03-27T06:14:06.801767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531520+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:04.531520+0000","last_peered":"2024-03-27T06:14:04.531520+0000","last_clean":"2024-03-27T06:14:04.531520+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:04.531520+0000","last_undegraded":"2024-03-27T06:14:04.531520+0000","last_fullsized":"2024-03-27T06:14:04.531520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529895+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:04.529895+0000","last_peered":"2024-03-27T06:14:04.529895+0000","last_clean":"2024-03-27T06:14:04.529895+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:04.529895+0000","last_undegraded":"2024-03-27T06:14:04.529895+0000","last_fullsized":"2024-03-27T06:14:04.529895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808793+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:07.808793+0000","last_peered":"2024-03-27T06:14:07.808793+0000","last_clean":"2024-03-27T06:14:07.808793+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:07.808793+0000","last_undegraded":"2024-03-27T06:14:07.808793+0000","last_fullsized":"2024-03-27T06:14:07.808793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527370+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:04.527370+0000","last_peered":"2024-03-27T06:14:04.527370+0000","last_clean":"2024-03-27T06:14:04.527370+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:04.527370+0000","last_undegraded":"2024-03-27T06:14:04.527370+0000","last_fullsized":"2024-03-27T06:14:04.527370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708423+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:06.708423+0000","last_peered":"2024-03-27T06:14:06.708423+0000","last_clean":"2024-03-27T06:14:06.708423+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:06.708423+0000","last_undegraded":"2024-03-27T06:14:06.708423+0000","last_fullsized":"2024-03-27T06:14:06.708423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531072+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:04.531072+0000","last_peered":"2024-03-27T06:14:04.531072+0000","last_clean":"2024-03-27T06:14:04.531072+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:04.531072+0000","last_undegraded":"2024-03-27T06:14:04.531072+0000","last_fullsized":"2024-03-27T06:14:04.531072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048444+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:07.048444+0000","last_peered":"2024-03-27T06:14:07.048444+0000","last_clean":"2024-03-27T06:14:07.048444+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:07.048444+0000","last_undegraded":"2024-03-27T06:14:07.048444+0000","last_fullsized":"2024-03-27T06:14:07.048444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708415+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:06.708415+0000","last_peered":"2024-03-27T06:14:06.708415+0000","last_clean":"2024-03-27T06:14:06.708415+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:06.708415+0000","last_undegraded":"2024-03-27T06:14:06.708415+0000","last_fullsized":"2024-03-27T06:14:06.708415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462059+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:07.462059+0000","last_peered":"2024-03-27T06:14:07.462059+0000","last_clean":"2024-03-27T06:14:07.462059+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:07.462059+0000","last_undegraded":"2024-03-27T06:14:07.462059+0000","last_fullsized":"2024-03-27T06:14:07.462059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530500+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:04.530500+0000","last_peered":"2024-03-27T06:14:04.530500+0000","last_clean":"2024-03-27T06:14:04.530500+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:04.530500+0000","last_undegraded":"2024-03-27T06:14:04.530500+0000","last_fullsized":"2024-03-27T06:14:04.530500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809116+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:07.809116+0000","last_peered":"2024-03-27T06:14:07.809116+0000","last_clean":"2024-03-27T06:14:07.809116+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:07.809116+0000","last_undegraded":"2024-03-27T06:14:07.809116+0000","last_fullsized":"2024-03-27T06:14:07.809116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046678+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:07.046678+0000","last_peered":"2024-03-27T06:14:07.046678+0000","last_clean":"2024-03-27T06:14:07.046678+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:07.046678+0000","last_undegraded":"2024-03-27T06:14:07.046678+0000","last_fullsized":"2024-03-27T06:14:07.046678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046680+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:07.046680+0000","last_peered":"2024-03-27T06:14:07.046680+0000","last_clean":"2024-03-27T06:14:07.046680+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:07.046680+0000","last_undegraded":"2024-03-27T06:14:07.046680+0000","last_fullsized":"2024-03-27T06:14:07.046680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.527343+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:08.527343+0000","last_peered":"2024-03-27T06:14:08.527343+0000","last_clean":"2024-03-27T06:14:08.527343+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:08.527343+0000","last_undegraded":"2024-03-27T06:14:08.527343+0000","last_fullsized":"2024-03-27T06:14:08.527343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046583+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:07.046583+0000","last_peered":"2024-03-27T06:14:07.046583+0000","last_clean":"2024-03-27T06:14:07.046583+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:07.046583+0000","last_undegraded":"2024-03-27T06:14:07.046583+0000","last_fullsized":"2024-03-27T06:14:07.046583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046901+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:07.046901+0000","last_peered":"2024-03-27T06:14:07.046901+0000","last_clean":"2024-03-27T06:14:07.046901+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:07.046901+0000","last_undegraded":"2024-03-27T06:14:07.046901+0000","last_fullsized":"2024-03-27T06:14:07.046901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046348+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:07.046348+0000","last_peered":"2024-03-27T06:14:07.046348+0000","last_clean":"2024-03-27T06:14:07.046348+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:07.046348+0000","last_undegraded":"2024-03-27T06:14:07.046348+0000","last_fullsized":"2024-03-27T06:14:07.046348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808963+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:07.808963+0000","last_peered":"2024-03-27T06:14:07.808963+0000","last_clean":"2024-03-27T06:14:07.808963+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:07.808963+0000","last_undegraded":"2024-03-27T06:14:07.808963+0000","last_fullsized":"2024-03-27T06:14:07.808963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.526612+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:08.526612+0000","last_peered":"2024-03-27T06:14:08.526612+0000","last_clean":"2024-03-27T06:14:08.526612+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:08.526612+0000","last_undegraded":"2024-03-27T06:14:08.526612+0000","last_fullsized":"2024-03-27T06:14:08.526612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525809+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:08.525809+0000","last_peered":"2024-03-27T06:14:08.525809+0000","last_clean":"2024-03-27T06:14:08.525809+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:08.525809+0000","last_undegraded":"2024-03-27T06:14:08.525809+0000","last_fullsized":"2024-03-27T06:14:08.525809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802159+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:06.802159+0000","last_peered":"2024-03-27T06:14:06.802159+0000","last_clean":"2024-03-27T06:14:06.802159+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:06.802159+0000","last_undegraded":"2024-03-27T06:14:06.802159+0000","last_fullsized":"2024-03-27T06:14:06.802159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525502+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:08.525502+0000","last_peered":"2024-03-27T06:14:08.525502+0000","last_clean":"2024-03-27T06:14:08.525502+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:08.525502+0000","last_undegraded":"2024-03-27T06:14:08.525502+0000","last_fullsized":"2024-03-27T06:14:08.525502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708697+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:06.708697+0000","last_peered":"2024-03-27T06:14:06.708697+0000","last_clean":"2024-03-27T06:14:06.708697+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:06.708697+0000","last_undegraded":"2024-03-27T06:14:06.708697+0000","last_fullsized":"2024-03-27T06:14:06.708697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708881+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:06.708881+0000","last_peered":"2024-03-27T06:14:06.708881+0000","last_clean":"2024-03-27T06:14:06.708881+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:06.708881+0000","last_undegraded":"2024-03-27T06:14:06.708881+0000","last_fullsized":"2024-03-27T06:14:06.708881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708688+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:06.708688+0000","last_peered":"2024-03-27T06:14:06.708688+0000","last_clean":"2024-03-27T06:14:06.708688+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:06.708688+0000","last_undegraded":"2024-03-27T06:14:06.708688+0000","last_fullsized":"2024-03-27T06:14:06.708688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525412+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:08.525412+0000","last_peered":"2024-03-27T06:14:08.525412+0000","last_clean":"2024-03-27T06:14:08.525412+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:08.525412+0000","last_undegraded":"2024-03-27T06:14:08.525412+0000","last_fullsized":"2024-03-27T06:14:08.525412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666065,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666043,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:14.171 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T06:14:14.171 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-27T06:14:14.171 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-27T06:14:14.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-27T06:14:14.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.422+0000 7f13c6200640 1 Processor -- start 2024-03-27T06:14:14.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.422+0000 7f13c6200640 1 -- start start 2024-03-27T06:14:14.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c6200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f13c006bb40 0x7f13c015e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c6200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 0x7f13c0154620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c6200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c015a960 con 0x7f13c006bf20 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c6200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f13c00cba40 con 0x7f13c006bb40 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c6200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f13c00cbc70 con 0x7f13c015b570 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13bfe00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f13c006bf20 0x7f13c015ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37802/0 (socket says 172.21.15.132:37802) 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13bfe00640 1 -- 172.21.15.132:0/2236768859 learned_addr learned my addr 172.21.15.132:0/2236768859 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c4e00640 1 --2- 172.21.15.132:0/2236768859 >> v2:172.21.15.132:3300/0 conn(0x7f13c006bb40 0x7f13c015e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13c5800640 1 --2- 172.21.15.132:0/2236768859 >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 0x7f13c0154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13bf400640 1 -- 172.21.15.132:0/2236768859 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158498033 0 0) 0x7f13c015a960 con 0x7f13c006bf20 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.424+0000 7f13bf400640 1 -- 172.21.15.132:0/2236768859 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13a4003520 con 0x7f13c006bf20 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.425+0000 7f13c4e00640 1 -- 172.21.15.132:0/2236768859 >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 msgr2=0x7f13c0154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.425+0000 7f13c4e00640 1 --2- 172.21.15.132:0/2236768859 >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 0x7f13c0154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.425+0000 7f13c4e00640 1 -- 172.21.15.132:0/2236768859 >> v1:172.21.15.134:6789/0 conn(0x7f13c006bf20 legacy=0x7f13c015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.425+0000 7f13c4e00640 1 -- 172.21.15.132:0/2236768859 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c00cbf50 con 0x7f13c006bb40 2024-03-27T06:14:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.425+0000 7f13c5800640 1 --2- 172.21.15.132:0/2236768859 >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 0x7f13c0154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13c4e00640 1 --2- 172.21.15.132:0/2236768859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c006bb40 0x7f13c015e960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f13c00b2ef0 tx=0x7f13b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae19ed95983f3d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13bf400640 1 -- 172.21.15.132:0/2236768859 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0039070 con 0x7f13c006bb40 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13bf400640 1 -- 172.21.15.132:0/2236768859 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b002cc80 con 0x7f13c006bb40 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13bf400640 1 -- 172.21.15.132:0/2236768859 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0034640 con 0x7f13c006bb40 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13c6200640 1 -- 172.21.15.132:0/2236768859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c006bb40 msgr2=0x7f13c015e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13c6200640 1 --2- 172.21.15.132:0/2236768859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c006bb40 0x7f13c015e960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f13c00b2ef0 tx=0x7f13b002bed0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.426+0000 7f13c6200640 1 -- 172.21.15.132:0/2236768859 shutdown_connections 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.427+0000 7f13c6200640 1 --2- 172.21.15.132:0/2236768859 >> v2:172.21.15.134:3300/0 conn(0x7f13c015b570 0x7f13c0154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.427+0000 7f13c6200640 1 --2- 172.21.15.132:0/2236768859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c006bb40 0x7f13c015e960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.427+0000 7f13c6200640 1 -- 172.21.15.132:0/2236768859 >> 172.21.15.132:0/2236768859 conn(0x7f13c006d910 msgr2=0x7f13c006dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.427+0000 7f13c6200640 1 -- 172.21.15.132:0/2236768859 shutdown_connections 2024-03-27T06:14:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.427+0000 7f13c6200640 1 -- 172.21.15.132:0/2236768859 wait complete. 2024-03-27T06:14:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.428+0000 7f13c6200640 1 Processor -- start 2024-03-27T06:14:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.428+0000 7f13c6200640 1 -- start start 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f13c006bb40 0x7f13c01f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c015b570 0x7f13c01e7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00cc7f0 con 0x7f13c015b570 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00cc510 con 0x7f13c006bf20 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c6200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f13c00ccc50 con 0x7f13c006bb40 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13bfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13c5800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c015b570 0x7f13c01e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13bfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56422/0 (socket says 172.21.15.132:56422) 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.429+0000 7f13bfe00640 1 -- 172.21.15.132:0/1866810131 learned_addr learned my addr 172.21.15.132:0/1866810131 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13c4e00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f13c006bb40 0x7f13c01f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 -- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f13c006bb40 msgr2=0x7f13c01f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f13c006bb40 0x7f13c01f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 -- 172.21.15.132:0/1866810131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c015b570 msgr2=0x7f13c01e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c015b570 0x7f13c01e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 -- 172.21.15.132:0/1866810131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c0159300 con 0x7f13c006bf20 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bfe00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c430 tx=0x7f13a800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8017070 con 0x7f13c006bf20 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8005d00 con 0x7f13c006bf20 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8007310 con 0x7f13c006bf20 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c015a2b0 con 0x7f13c006bf20 2024-03-27T06:14:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.430+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c0150900 con 0x7f13c006bf20 2024-03-27T06:14:14.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.431+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c015a7d0 con 0x7f13c006bf20 2024-03-27T06:14:14.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.432+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f13a801a070 con 0x7f13c006bf20 2024-03-27T06:14:14.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.433+0000 7f13bcc00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 0x7f139c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.433+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f13a808a9b0 con 0x7f13c006bf20 2024-03-27T06:14:14.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.433+0000 7f13c4e00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 0x7f139c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.434+0000 7f13c4e00640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 0x7f139c06e9f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036000 tx=0x7f13b0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.438+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f13a8058970 con 0x7f13c006bf20 2024-03-27T06:14:14.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.658+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f13c0068730 con 0x7f13c006bf20 2024-03-27T06:14:14.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.659+0000 7f13bcc00640 1 -- 172.21.15.132:0/1866810131 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f13a805cc70 con 0x7f13c006bf20 2024-03-27T06:14:14.650 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:14.650 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 msgr2=0x7f139c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 0x7f139c06e9f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036000 tx=0x7f13b0030000 comp rx=0 tx=0).stop 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 msgr2=0x7f13c01503c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c430 tx=0x7f13a800c830 comp rx=0 tx=0).stop 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 shutdown_connections 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c015b570 0x7f13c01e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f139c06c530 0x7f139c06e9f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13c006bf20 0x7f13c01503c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 --2- 172.21.15.132:0/1866810131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f13c006bb40 0x7f13c01f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 >> 172.21.15.132:0/1866810131 conn(0x7f13c006d910 msgr2=0x7f13c00cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:14.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 shutdown_connections 2024-03-27T06:14:14.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.662+0000 7f13c6200640 1 -- 172.21.15.132:0/1866810131 wait complete. 2024-03-27T06:14:14.671 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-27T06:14:14.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:14.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.883+0000 7f8a75a00640 1 Processor -- start 2024-03-27T06:14:14.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.883+0000 7f8a75a00640 1 -- start start 2024-03-27T06:14:14.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.884+0000 7f8a75a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70162e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.884+0000 7f8a75a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7006b520 con 0x7f8a701516f0 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.884+0000 7f8a75a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7006b7c0 con 0x7f8a70151ad0 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.884+0000 7f8a75a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8a70072be0 con 0x7f8a70157c80 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a66000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8a70151ad0 0x7f8a70157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37812/0 (socket says 172.21.15.132:37812) 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a66000640 1 -- 172.21.15.132:0/554015699 learned_addr learned my addr 172.21.15.132:0/554015699 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6f400640 1 --2- 172.21.15.132:0/554015699 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843279734 0 0) 0x7f8a7006b7c0 con 0x7f8a70151ad0 2024-03-27T06:14:14.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a54005730 con 0x7f8a70151ad0 2024-03-27T06:14:14.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020887139 0 0) 0x7f8a7006b520 con 0x7f8a701516f0 2024-03-27T06:14:14.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a7006b7c0 con 0x7f8a701516f0 2024-03-27T06:14:14.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6f400640 1 -- 172.21.15.132:0/554015699 >> v1:172.21.15.134:6789/0 conn(0x7f8a70151ad0 legacy=0x7f8a70157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6f400640 1 -- 172.21.15.132:0/554015699 >> v1:172.21.15.132:6789/0 conn(0x7f8a701516f0 legacy=0x7f8a700c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.885+0000 7f8a6f400640 1 -- 172.21.15.132:0/554015699 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7006baa0 con 0x7f8a70157c80 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.886+0000 7f8a6f400640 1 --2- 172.21.15.132:0/554015699 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70162e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a70072d60 tx=0x7f8a6802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb29ddb1e68feecf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68039070 con 0x7f8a70157c80 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6802cd50 con 0x7f8a70157c80 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a6e000640 1 -- 172.21.15.132:0/554015699 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68034660 con 0x7f8a70157c80 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 -- 172.21.15.132:0/554015699 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 msgr2=0x7f8a70162e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 --2- 172.21.15.132:0/554015699 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70162e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a70072d60 tx=0x7f8a6802bfa0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 -- 172.21.15.132:0/554015699 shutdown_connections 2024-03-27T06:14:14.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 --2- 172.21.15.132:0/554015699 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70162e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 -- 172.21.15.132:0/554015699 >> 172.21.15.132:0/554015699 conn(0x7f8a700cc060 msgr2=0x7f8a700cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:14.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.887+0000 7f8a75a00640 1 -- 172.21.15.132:0/554015699 shutdown_connections 2024-03-27T06:14:14.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.888+0000 7f8a75a00640 1 -- 172.21.15.132:0/554015699 wait complete. 2024-03-27T06:14:14.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.889+0000 7f8a75a00640 1 Processor -- start 2024-03-27T06:14:14.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.889+0000 7f8a75a00640 1 -- start start 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 0x7f8a700cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a70151ad0 0x7f8a70155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7006bd80 con 0x7f8a70151ad0 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70072870 con 0x7f8a701516f0 2024-03-27T06:14:14.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a75a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8a700c1080 con 0x7f8a70157c80 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a6ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 0x7f8a700cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a6ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 0x7f8a700cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56434/0 (socket says 172.21.15.132:56434) 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a66000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a70151ad0 0x7f8a70155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a6ea00640 1 -- 172.21.15.132:0/3812568807 learned_addr learned my addr 172.21.15.132:0/3812568807 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.890+0000 7f8a6f400640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 -- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 msgr2=0x7f8a700cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 0x7f8a700cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 -- 172.21.15.132:0/3812568807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a70151ad0 msgr2=0x7f8a70155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a70151ad0 0x7f8a70155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 -- 172.21.15.132:0/3812568807 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700c5520 con 0x7f8a70157c80 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a6f400640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70145930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a7006b6f0 tx=0x7f8a68004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68039050 con 0x7f8a70157c80 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68045070 con 0x7f8a70157c80 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a700c5800 con 0x7f8a70157c80 2024-03-27T06:14:14.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.891+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68009450 con 0x7f8a70157c80 2024-03-27T06:14:14.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.892+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a700c5dc0 con 0x7f8a70157c80 2024-03-27T06:14:14.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.893+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a700c61a0 con 0x7f8a70157c80 2024-03-27T06:14:14.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.893+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8a6803e850 con 0x7f8a70157c80 2024-03-27T06:14:14.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.894+0000 7f8a67400640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 0x7f8a4c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:14.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.895+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8a680ae270 con 0x7f8a70157c80 2024-03-27T06:14:14.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.895+0000 7f8a6ea00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 0x7f8a4c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:14.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.895+0000 7f8a6ea00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 0x7f8a4c06e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:14.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:14.899+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f8a700c61a0 con 0x7f8a70157c80 2024-03-27T06:14:15.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.098+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a70068840 con 0x7f8a70157c80 2024-03-27T06:14:15.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.100+0000 7f8a67400640 1 -- 172.21.15.132:0/3812568807 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f8a6807c230 con 0x7f8a70157c80 2024-03-27T06:14:15.091 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:15.091 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:15.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.102+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 msgr2=0x7f8a4c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.102+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 0x7f8a4c06e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60009040 comp rx=0 tx=0).stop 2024-03-27T06:14:15.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.102+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 msgr2=0x7f8a70145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.102+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70145930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a7006b6f0 tx=0x7f8a68004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 shutdown_connections 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8a70157c80 0x7f8a70145930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a70151ad0 0x7f8a70155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8a4c06c530 0x7f8a4c06e9f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 --2- 172.21.15.132:0/3812568807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a701516f0 0x7f8a700cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 >> 172.21.15.132:0/3812568807 conn(0x7f8a700cc060 msgr2=0x7f8a700cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 shutdown_connections 2024-03-27T06:14:15.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.103+0000 7f8a75a00640 1 -- 172.21.15.132:0/3812568807 wait complete. 2024-03-27T06:14:15.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:14:15.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:14:15.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.324+0000 7f9193000640 1 Processor -- start 2024-03-27T06:14:15.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.325+0000 7f9193000640 1 -- start start 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f9193000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f918c15c380 0x7f918c15c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f9193000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f918c071930 con 0x7f918c15bfa0 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f9193000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f918c071bd0 con 0x7f918c152500 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f9193000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f918c072110 con 0x7f918c15c380 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f918be00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f918c15bfa0 0x7f918c0735f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33072/0 (socket says 172.21.15.132:33072) 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f918be00640 1 -- 172.21.15.132:0/3276843550 learned_addr learned my addr 172.21.15.132:0/3276843550 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:15.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.326+0000 7f9183400640 1 --2- 172.21.15.132:0/3276843550 >> v2:172.21.15.134:3300/0 conn(0x7f918c15c380 0x7f918c15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780060605 0 0) 0x7f918c071bd0 con 0x7f918c152500 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9170003520 con 0x7f918c152500 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100760028 0 0) 0x7f918c071930 con 0x7f918c15bfa0 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f9183400640 1 -- 172.21.15.132:0/3276843550 >> v1:172.21.15.134:6789/0 conn(0x7f918c152500 legacy=0x7f918c06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f9183400640 1 -- 172.21.15.132:0/3276843550 >> v1:172.21.15.132:6789/0 conn(0x7f918c15bfa0 legacy=0x7f918c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.327+0000 7f9183400640 1 -- 172.21.15.132:0/3276843550 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c071e70 con 0x7f918c15c380 2024-03-27T06:14:15.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.328+0000 7f9183400640 1 --2- 172.21.15.132:0/3276843550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c15c910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91740097a0 tx=0x7f917402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcdd18363b4fb22c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.328+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917402f810 con 0x7f918c15c380 2024-03-27T06:14:15.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.328+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917402f9b0 con 0x7f918c15c380 2024-03-27T06:14:15.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.328+0000 7f918b400640 1 -- 172.21.15.132:0/3276843550 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917402fd80 con 0x7f918c15c380 2024-03-27T06:14:15.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 -- 172.21.15.132:0/3276843550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 msgr2=0x7f918c15c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 --2- 172.21.15.132:0/3276843550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c15c910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91740097a0 tx=0x7f917402b9f0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 -- 172.21.15.132:0/3276843550 shutdown_connections 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 --2- 172.21.15.132:0/3276843550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c15c910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 -- 172.21.15.132:0/3276843550 >> 172.21.15.132:0/3276843550 conn(0x7f918c0c89a0 msgr2=0x7f918c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 -- 172.21.15.132:0/3276843550 shutdown_connections 2024-03-27T06:14:15.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.329+0000 7f9193000640 1 -- 172.21.15.132:0/3276843550 wait complete. 2024-03-27T06:14:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.331+0000 7f9193000640 1 Processor -- start 2024-03-27T06:14:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.332+0000 7f9193000640 1 -- start start 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 0x7f918c0c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f918c0c97e0 con 0x7f918c152500 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06c530 con 0x7f918c15c380 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9193000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f918c06ca60 con 0x7f918c15bfa0 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f918be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f918be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37540/0 (socket says 172.21.15.132:37540) 2024-03-27T06:14:15.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f918be00640 1 -- 172.21.15.132:0/4014196334 learned_addr learned my addr 172.21.15.132:0/4014196334 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9191000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.333+0000 7f9183400640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 0x7f918c0c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 -- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 msgr2=0x7f918c0c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 0x7f918c0c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 -- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 msgr2=0x7f918c148680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c148680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 -- 172.21.15.132:0/4014196334 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c15b390 con 0x7f918c152500 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9191000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9183400640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 0x7f918c0c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f918be00640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c480 tx=0x7f917c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c005ce0 con 0x7f918c152500 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c007070 con 0x7f918c152500 2024-03-27T06:14:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c007460 con 0x7f918c152500 2024-03-27T06:14:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918c0c3fb0 con 0x7f918c152500 2024-03-27T06:14:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.334+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f918c0ca120 con 0x7f918c152500 2024-03-27T06:14:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.336+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f918c068950 con 0x7f918c152500 2024-03-27T06:14:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.336+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f917c007600 con 0x7f918c152500 2024-03-27T06:14:15.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.338+0000 7f9188c00640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 0x7f916806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.339+0000 7f9183400640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 0x7f916806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.339+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f917c08b1a0 con 0x7f918c152500 2024-03-27T06:14:15.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.339+0000 7f9183400640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 0x7f916806e9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9174036000 tx=0x7f9174008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.343+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f917c059200 con 0x7f918c152500 2024-03-27T06:14:15.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.542+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f918c156160 con 0x7f918c152500 2024-03-27T06:14:15.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.543+0000 7f9188c00640 1 -- 172.21.15.132:0/4014196334 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f917c05d500 con 0x7f918c152500 2024-03-27T06:14:15.534 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 msgr2=0x7f916806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 0x7f916806e9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9174036000 tx=0x7f9174008030 comp rx=0 tx=0).stop 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 msgr2=0x7f918c0c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c480 tx=0x7f917c00c940 comp rx=0 tx=0).stop 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 shutdown_connections 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f916806c530 0x7f916806e9f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f918c15c380 0x7f918c148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f918c15bfa0 0x7f918c0c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.537 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 --2- 172.21.15.132:0/4014196334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f918c152500 0x7f918c0c22b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 >> 172.21.15.132:0/4014196334 conn(0x7f918c0c89a0 msgr2=0x7f918c150a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:15.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 shutdown_connections 2024-03-27T06:14:15.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.546+0000 7f9193000640 1 -- 172.21.15.132:0/4014196334 wait complete. 2024-03-27T06:14:15.554 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:14:15.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.774+0000 7f2f7ee00640 1 Processor -- start 2024-03-27T06:14:15.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.774+0000 7f2f7ee00640 1 -- start start 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f7ee00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f7ee00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2f780c9570 0x7f2f7815e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f7ee00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f780c6ec0 con 0x7f2f7815eb50 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f7ee00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2f780c1230 con 0x7f2f780c9570 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f7ee00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2f780c7180 con 0x7f2f7815e230 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f77e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f77400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2f780c9570 0x7f2f7815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f77e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35414/0 (socket says 172.21.15.132:35414) 2024-03-27T06:14:15.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.776+0000 7f2f77e00640 1 -- 172.21.15.132:0/1645811139 learned_addr learned my addr 172.21.15.132:0/1645811139 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:15.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.777+0000 7f2f77e00640 1 -- 172.21.15.132:0/1645811139 >> v2:172.21.15.132:3300/0 conn(0x7f2f780c9570 msgr2=0x7f2f7815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:15.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.777+0000 7f2f77e00640 1 --2- 172.21.15.132:0/1645811139 >> v2:172.21.15.132:3300/0 conn(0x7f2f780c9570 0x7f2f7815e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.777+0000 7f2f77e00640 1 -- 172.21.15.132:0/1645811139 >> v1:172.21.15.132:6789/0 conn(0x7f2f7815eb50 legacy=0x7f2f78157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.777+0000 7f2f77e00640 1 -- 172.21.15.132:0/1645811139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f780c7460 con 0x7f2f7815e230 2024-03-27T06:14:15.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.778+0000 7f2f77e00640 1 --2- 172.21.15.132:0/1645811139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009870 tx=0x7f2f6802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423504c344c473e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.778+0000 7f2f76a00640 1 -- 172.21.15.132:0/1645811139 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6802f840 con 0x7f2f7815e230 2024-03-27T06:14:15.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.778+0000 7f2f76a00640 1 -- 172.21.15.132:0/1645811139 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f6802f9e0 con 0x7f2f7815e230 2024-03-27T06:14:15.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.778+0000 7f2f76a00640 1 -- 172.21.15.132:0/1645811139 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6802fdb0 con 0x7f2f7815e230 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 -- 172.21.15.132:0/1645811139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 msgr2=0x7f2f780c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/1645811139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009870 tx=0x7f2f6802ba20 comp rx=0 tx=0).stop 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 -- 172.21.15.132:0/1645811139 shutdown_connections 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/1645811139 >> v2:172.21.15.132:3300/0 conn(0x7f2f780c9570 0x7f2f7815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/1645811139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f780c68a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 -- 172.21.15.132:0/1645811139 >> 172.21.15.132:0/1645811139 conn(0x7f2f780c89a0 msgr2=0x7f2f780c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 -- 172.21.15.132:0/1645811139 shutdown_connections 2024-03-27T06:14:15.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.779+0000 7f2f7ee00640 1 -- 172.21.15.132:0/1645811139 wait complete. 2024-03-27T06:14:15.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.781+0000 7f2f7ee00640 1 Processor -- start 2024-03-27T06:14:15.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.781+0000 7f2f7ee00640 1 -- start start 2024-03-27T06:14:15.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f781f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 0x7f2f78150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78158260 con 0x7f2f7815eb50 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2f780c0ec0 con 0x7f2f780c9570 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.782+0000 7f2f7ee00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2f781583e0 con 0x7f2f7815e230 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f781f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56460/0 (socket says 172.21.15.132:56460) 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 -- 172.21.15.132:0/2076114232 learned_addr learned my addr 172.21.15.132:0/2076114232 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f7ce00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 0x7f2f78150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 -- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 msgr2=0x7f2f781f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f781f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 -- 172.21.15.132:0/2076114232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 msgr2=0x7f2f78150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 0x7f2f78150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 -- 172.21.15.132:0/2076114232 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f780c4460 con 0x7f2f780c9570 2024-03-27T06:14:15.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f7ce00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 0x7f2f78150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f77e00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f68000c00 tx=0x7f2f68004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.783+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68037bc0 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.784+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f68046070 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.784+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f780c4740 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.784+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68041400 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.784+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f780c4d00 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.785+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f780c3220 con 0x7f2f780c9570 2024-03-27T06:14:15.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.785+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2f68040070 con 0x7f2f780c9570 2024-03-27T06:14:15.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.786+0000 7f2f6fe00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 0x7f2f5806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:15.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.787+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2f680add00 con 0x7f2f780c9570 2024-03-27T06:14:15.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.787+0000 7f2f77400640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 0x7f2f5806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:15.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.787+0000 7f2f77400640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 0x7f2f5806ea20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f780aef30 tx=0x7f2f64009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:15.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.791+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2f68033030 con 0x7f2f780c9570 2024-03-27T06:14:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.989+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2f78156b00 con 0x7f2f780c9570 2024-03-27T06:14:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.990+0000 7f2f6fe00640 1 -- 172.21.15.132:0/2076114232 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f2f78156b00 con 0x7f2f780c9570 2024-03-27T06:14:15.981 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 msgr2=0x7f2f5806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 0x7f2f5806ea20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f780aef30 tx=0x7f2f64009190 comp rx=0 tx=0).stop 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 msgr2=0x7f2f781509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f68000c00 tx=0x7f2f68004730 comp rx=0 tx=0).stop 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 shutdown_connections 2024-03-27T06:14:15.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f7815eb50 0x7f2f78150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2f7815e230 0x7f2f781f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2f5806c560 0x7f2f5806ea20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 --2- 172.21.15.132:0/2076114232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f780c9570 0x7f2f781509b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 >> 172.21.15.132:0/2076114232 conn(0x7f2f780c89a0 msgr2=0x7f2f7806ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 shutdown_connections 2024-03-27T06:14:15.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:15.993+0000 7f2f7ee00640 1 -- 172.21.15.132:0/2076114232 wait complete. 2024-03-27T06:14:16.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:14:16.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.228+0000 7ffb73600640 1 Processor -- start 2024-03-27T06:14:16.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.228+0000 7ffb73600640 1 -- start start 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb73600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c0c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb73600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6c06c530 con 0x7ffb6c158360 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb73600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6c06c7d0 con 0x7ffb6c151bf0 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb73600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6c06cab0 con 0x7ffb6c151810 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb70c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c0c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb63e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffb6c151bf0 0x7ffb6c072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37830/0 (socket says 172.21.15.132:37830) 2024-03-27T06:14:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.230+0000 7ffb63e00640 1 -- 172.21.15.132:0/3026589423 learned_addr learned my addr 172.21.15.132:0/3026589423 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.231+0000 7ffb70c00640 1 -- 172.21.15.132:0/3026589423 >> v1:172.21.15.134:6789/0 conn(0x7ffb6c151bf0 legacy=0x7ffb6c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.231+0000 7ffb6be00640 1 -- 172.21.15.132:0/3026589423 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 719855182 0 0) 0x7ffb6c06c7d0 con 0x7ffb6c151bf0 2024-03-27T06:14:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.231+0000 7ffb70c00640 1 -- 172.21.15.132:0/3026589423 >> v1:172.21.15.132:6789/0 conn(0x7ffb6c158360 legacy=0x7ffb6c072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.231+0000 7ffb70c00640 1 -- 172.21.15.132:0/3026589423 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c157880 con 0x7ffb6c151810 2024-03-27T06:14:16.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.232+0000 7ffb70c00640 1 --2- 172.21.15.132:0/3026589423 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c0c1d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0097d0 tx=0x7ffb5c02dbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c19a9a8813d0527 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.232+0000 7ffb6be00640 1 -- 172.21.15.132:0/3026589423 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c03b070 con 0x7ffb6c151810 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.232+0000 7ffb6be00640 1 -- 172.21.15.132:0/3026589423 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c02e980 con 0x7ffb6c151810 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.232+0000 7ffb6be00640 1 -- 172.21.15.132:0/3026589423 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c02ed50 con 0x7ffb6c151810 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 -- 172.21.15.132:0/3026589423 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 msgr2=0x7ffb6c0c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 --2- 172.21.15.132:0/3026589423 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c0c1d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0097d0 tx=0x7ffb5c02dbd0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 -- 172.21.15.132:0/3026589423 shutdown_connections 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 --2- 172.21.15.132:0/3026589423 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c0c1d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 -- 172.21.15.132:0/3026589423 >> 172.21.15.132:0/3026589423 conn(0x7ffb6c0c89a0 msgr2=0x7ffb6c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 -- 172.21.15.132:0/3026589423 shutdown_connections 2024-03-27T06:14:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.233+0000 7ffb73600640 1 -- 172.21.15.132:0/3026589423 wait complete. 2024-03-27T06:14:16.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.234+0000 7ffb73600640 1 Processor -- start 2024-03-27T06:14:16.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.235+0000 7ffb73600640 1 -- start start 2024-03-27T06:14:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.235+0000 7ffb73600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.235+0000 7ffb73600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb6c158360 0x7ffb6c150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb73600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c154b30 0x7ffb6c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb73600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c0c22c0 con 0x7ffb6c158360 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb73600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c157e40 con 0x7ffb6c154b30 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb73600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6c0c86d0 con 0x7ffb6c151810 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35448/0 (socket says 172.21.15.132:35448) 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 -- 172.21.15.132:0/3879077381 learned_addr learned my addr 172.21.15.132:0/3879077381 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb63e00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb6c158360 0x7ffb6c150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb71600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c154b30 0x7ffb6c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 -- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c154b30 msgr2=0x7ffb6c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c154b30 0x7ffb6c153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 -- 172.21.15.132:0/3879077381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb6c158360 msgr2=0x7ffb6c150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb6c158360 0x7ffb6c150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 -- 172.21.15.132:0/3879077381 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c144290 con 0x7ffb6c151810 2024-03-27T06:14:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.236+0000 7ffb70c00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0098f0 tx=0x7ffb5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.237+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c03b070 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.237+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c041410 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.237+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c041800 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.237+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c1522d0 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.237+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c1565a0 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.238+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb6c156390 con 0x7ffb6c151810 2024-03-27T06:14:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.239+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ffb5c048060 con 0x7ffb6c151810 2024-03-27T06:14:16.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.239+0000 7ffb69600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 0x7ffb4806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.240+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffb5c0aceb0 con 0x7ffb6c151810 2024-03-27T06:14:16.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.240+0000 7ffb63e00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 0x7ffb4806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.240+0000 7ffb63e00640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 0x7ffb4806ea20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007f70 tx=0x7ffb54007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.244+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ffb5c07ad50 con 0x7ffb6c151810 2024-03-27T06:14:16.382 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:14:16.391+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-27T06:14:16.382 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:14:16.391+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-27T06:14:16.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.442+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffb6c068950 con 0x7ffb6c151810 2024-03-27T06:14:16.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.442+0000 7ffb69600640 1 -- 172.21.15.132:0/3879077381 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7ffb5c07f050 con 0x7ffb6c151810 2024-03-27T06:14:16.433 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:14:16.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 msgr2=0x7ffb4806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 0x7ffb4806ea20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007f70 tx=0x7ffb54007f00 comp rx=0 tx=0).stop 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 msgr2=0x7ffb6c143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0098f0 tx=0x7ffb5c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 shutdown_connections 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb4806c560 0x7ffb4806ea20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c154b30 0x7ffb6c153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb6c158360 0x7ffb6c150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 --2- 172.21.15.132:0/3879077381 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffb6c151810 0x7ffb6c143810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 >> 172.21.15.132:0/3879077381 conn(0x7ffb6c0c89a0 msgr2=0x7ffb6c06d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:16.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 shutdown_connections 2024-03-27T06:14:16.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.445+0000 7ffb73600640 1 -- 172.21.15.132:0/3879077381 wait complete. 2024-03-27T06:14:16.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:16.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.670+0000 7f4c29c00640 1 Processor -- start 2024-03-27T06:14:16.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.670+0000 7f4c29c00640 1 -- start start 2024-03-27T06:14:16.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.671+0000 7f4c29c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.671+0000 7f4c29c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4c24073970 0x7f4c2406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.671+0000 7f4c29c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2406b630 con 0x7f4c24073550 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.671+0000 7f4c29c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4c24152d30 con 0x7f4c24073970 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c29c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2406b8d0 con 0x7f4c24152950 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c23400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4c24073970 0x7f4c2406cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35454/0 (socket says 172.21.15.132:35454) 2024-03-27T06:14:16.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 -- 172.21.15.132:0/4081174714 learned_addr learned my addr 172.21.15.132:0/4081174714 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:16.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 -- 172.21.15.132:0/4081174714 >> v2:172.21.15.132:3300/0 conn(0x7f4c24073970 msgr2=0x7f4c2406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:16.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 --2- 172.21.15.132:0/4081174714 >> v2:172.21.15.132:3300/0 conn(0x7f4c24073970 0x7f4c2406cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 -- 172.21.15.132:0/4081174714 >> v1:172.21.15.132:6789/0 conn(0x7f4c24073550 legacy=0x7f4c24071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.672+0000 7f4c22a00640 1 -- 172.21.15.132:0/4081174714 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2406bbb0 con 0x7f4c24152950 2024-03-27T06:14:16.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.673+0000 7f4c22a00640 1 --2- 172.21.15.132:0/4081174714 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c1402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f081b47fe4e0be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c22000640 1 -- 172.21.15.132:0/4081174714 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14034070 con 0x7f4c24152950 2024-03-27T06:14:16.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c22000640 1 -- 172.21.15.132:0/4081174714 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c14030840 con 0x7f4c24152950 2024-03-27T06:14:16.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c22000640 1 -- 172.21.15.132:0/4081174714 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14030c10 con 0x7f4c24152950 2024-03-27T06:14:16.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 -- 172.21.15.132:0/4081174714 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 msgr2=0x7f4c241590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 --2- 172.21.15.132:0/4081174714 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c1402b980 comp rx=0 tx=0).stop 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 -- 172.21.15.132:0/4081174714 shutdown_connections 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 --2- 172.21.15.132:0/4081174714 >> v2:172.21.15.132:3300/0 conn(0x7f4c24073970 0x7f4c2406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 --2- 172.21.15.132:0/4081174714 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24152950 0x7f4c241590a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 -- 172.21.15.132:0/4081174714 >> 172.21.15.132:0/4081174714 conn(0x7f4c240cc060 msgr2=0x7f4c240cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.674+0000 7f4c29c00640 1 -- 172.21.15.132:0/4081174714 shutdown_connections 2024-03-27T06:14:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.675+0000 7f4c29c00640 1 -- 172.21.15.132:0/4081174714 wait complete. 2024-03-27T06:14:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.676+0000 7f4c29c00640 1 Processor -- start 2024-03-27T06:14:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.676+0000 7f4c29c00640 1 -- start start 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 0x7f4c240ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2406c450 con 0x7f4c24152950 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2406c170 con 0x7f4c24073550 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c29c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4c24071880 con 0x7f4c24073970 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c1a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c22a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c23400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 0x7f4c240ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c1a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35468/0 (socket says 172.21.15.132:35468) 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c22a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56468/0 (socket says 172.21.15.132:56468) 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c23400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 0x7f4c240ae8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37582/0 (socket says 172.21.15.132:37582) 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.677+0000 7f4c1a000640 1 -- 172.21.15.132:0/2994739055 learned_addr learned my addr 172.21.15.132:0/2994739055 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 -- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 msgr2=0x7f4c240af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 -- 172.21.15.132:0/2994739055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 msgr2=0x7f4c240ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 0x7f4c240ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 -- 172.21.15.132:0/2994739055 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c240c2b80 con 0x7f4c24073970 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c22a00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1a000640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00c400 tx=0x7f4c0c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c0052d0 con 0x7f4c24073970 2024-03-27T06:14:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c0c005490 con 0x7f4c24073970 2024-03-27T06:14:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c24159750 con 0x7f4c24073970 2024-03-27T06:14:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.678+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c007260 con 0x7f4c24073970 2024-03-27T06:14:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.679+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c241f0390 con 0x7f4c24073970 2024-03-27T06:14:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.680+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c24072df0 con 0x7f4c24073970 2024-03-27T06:14:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.680+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4c0c007400 con 0x7f4c24073970 2024-03-27T06:14:16.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.681+0000 7f4c1b400640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 0x7f4c0006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:16.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.681+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4c0c08abc0 con 0x7f4c24073970 2024-03-27T06:14:16.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.681+0000 7f4c22a00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 0x7f4c0006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:16.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.682+0000 7f4c22a00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 0x7f4c0006e9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c14036000 tx=0x7f4c1403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:16.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.685+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4c24072df0 con 0x7f4c24073970 2024-03-27T06:14:16.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.923+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c241f0180 con 0x7f4c24073970 2024-03-27T06:14:16.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.924+0000 7f4c1b400640 1 -- 172.21.15.132:0/2994739055 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6877 (secure 0 0 0) 0x7f4c0c058b80 con 0x7f4c24073970 2024-03-27T06:14:16.915 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:16.915 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:13.590498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:14:16.915 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 11 2024-03-27T06:14:16.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 msgr2=0x7f4c0006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 0x7f4c0006e9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c14036000 tx=0x7f4c1403b040 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 msgr2=0x7f4c2414d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00c400 tx=0x7f4c0c00c8c0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 shutdown_connections 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c24152950 0x7f4c240ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4c24073970 0x7f4c2414d870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4c0006c530 0x7f4c0006e9f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 --2- 172.21.15.132:0/2994739055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c24073550 0x7f4c240af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.926+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 >> 172.21.15.132:0/2994739055 conn(0x7f4c240cc060 msgr2=0x7f4c240c8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.927+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 shutdown_connections 2024-03-27T06:14:16.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:16.927+0000 7f4c29c00640 1 -- 172.21.15.132:0/2994739055 wait complete. 2024-03-27T06:14:16.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:17.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.148+0000 7fae48200640 1 Processor -- start 2024-03-27T06:14:17.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.148+0000 7fae48200640 1 -- start start 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.149+0000 7fae48200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 0x7fae40071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae48200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae4006b630 con 0x7fae400716b0 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae48200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae4006b8d0 con 0x7fae40152950 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae48200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fae40152d30 con 0x7fae401590a0 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae45800640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fae40152950 0x7fae40073550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37842/0 (socket says 172.21.15.132:37842) 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae45800640 1 -- 172.21.15.132:0/1606243727 learned_addr learned my addr 172.21.15.132:0/1606243727 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae44e00640 1 --2- 172.21.15.132:0/1606243727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 0x7fae40071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486072968 0 0) 0x7fae4006b8d0 con 0x7fae40152950 2024-03-27T06:14:17.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae28003520 con 0x7fae40152950 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae44e00640 1 -- 172.21.15.132:0/1606243727 >> v1:172.21.15.134:6789/0 conn(0x7fae40152950 legacy=0x7fae40073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae44e00640 1 -- 172.21.15.132:0/1606243727 >> v1:172.21.15.132:6789/0 conn(0x7fae400716b0 legacy=0x7fae4006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741687826 0 0) 0x7fae4006b630 con 0x7fae400716b0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.150+0000 7fae44e00640 1 -- 172.21.15.132:0/1606243727 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4006bbb0 con 0x7fae401590a0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.151+0000 7fae44e00640 1 --2- 172.21.15.132:0/1606243727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 0x7fae40071170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0097d0 tx=0x7fae2c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=784f687a696ca690 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c035020 con 0x7fae401590a0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c031890 con 0x7fae401590a0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae3fe00640 1 -- 172.21.15.132:0/1606243727 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c031c60 con 0x7fae401590a0 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 -- 172.21.15.132:0/1606243727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 msgr2=0x7fae40071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 --2- 172.21.15.132:0/1606243727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 0x7fae40071170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0097d0 tx=0x7fae2c02b930 comp rx=0 tx=0).stop 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 -- 172.21.15.132:0/1606243727 shutdown_connections 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 --2- 172.21.15.132:0/1606243727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae401590a0 0x7fae40071170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 -- 172.21.15.132:0/1606243727 >> 172.21.15.132:0/1606243727 conn(0x7fae400cc060 msgr2=0x7fae400cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 -- 172.21.15.132:0/1606243727 shutdown_connections 2024-03-27T06:14:17.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.152+0000 7fae48200640 1 -- 172.21.15.132:0/1606243727 wait complete. 2024-03-27T06:14:17.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.154+0000 7fae48200640 1 Processor -- start 2024-03-27T06:14:17.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.154+0000 7fae48200640 1 -- start start 2024-03-27T06:14:17.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae400716b0 0x7fae400af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae401590a0 0x7fae400b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fae4006c450 con 0x7fae400716b0 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fae4006c170 con 0x7fae401590a0 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae48200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fae401594c0 con 0x7fae40152950 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae44e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae45800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae400716b0 0x7fae400af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae44e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35492/0 (socket says 172.21.15.132:35492) 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae44e00640 1 -- 172.21.15.132:0/2283071670 learned_addr learned my addr 172.21.15.132:0/2283071670 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.155+0000 7fae46200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae401590a0 0x7fae400b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 -- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae401590a0 msgr2=0x7fae400b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae401590a0 0x7fae400b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 -- 172.21.15.132:0/2283071670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae400716b0 msgr2=0x7fae400af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae400716b0 0x7fae400af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 -- 172.21.15.132:0/2283071670 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae400c5660 con 0x7fae40152950 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae44e00640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae2c034040 tx=0x7fae2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c040070 con 0x7fae40152950 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c030ca0 con 0x7fae40152950 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c035040 con 0x7fae40152950 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae401557f0 con 0x7fae40152950 2024-03-27T06:14:17.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.156+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae40153060 con 0x7fae40152950 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.157+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae40068840 con 0x7fae40152950 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.158+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fae2c038020 con 0x7fae40152950 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.159+0000 7fae3d600640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 0x7fae2006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.159+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae2c0ad3a0 con 0x7fae40152950 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.159+0000 7fae45800640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 0x7fae2006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.159+0000 7fae45800640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 0x7fae2006ea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fae34005880 tx=0x7fae34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.163+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fae2c02d040 con 0x7fae40152950 2024-03-27T06:14:17.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.349+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae400c6b30 con 0x7fae40152950 2024-03-27T06:14:17.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.351+0000 7fae3d600640 1 -- 172.21.15.132:0/2283071670 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fae2c07b240 con 0x7fae40152950 2024-03-27T06:14:17.342 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:17.342 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:17.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.353+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 msgr2=0x7fae2006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.353+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 0x7fae2006ea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fae34005880 tx=0x7fae34009040 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.353+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 msgr2=0x7fae400b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.353+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae2c034040 tx=0x7fae2c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 shutdown_connections 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae2006c560 0x7fae2006ea20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae401590a0 0x7fae400b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae40152950 0x7fae400b0370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 --2- 172.21.15.132:0/2283071670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae400716b0 0x7fae400af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 >> 172.21.15.132:0/2283071670 conn(0x7fae400cc060 msgr2=0x7fae400cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 shutdown_connections 2024-03-27T06:14:17.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.354+0000 7fae48200640 1 -- 172.21.15.132:0/2283071670 wait complete. 2024-03-27T06:14:17.363 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:14:17.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:14:17.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.577+0000 7fc0ee600640 1 Processor -- start 2024-03-27T06:14:17.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.577+0000 7fc0ee600640 1 -- start start 2024-03-27T06:14:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.578+0000 7fc0ee600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc0e8151bf0 0x7fc0e8072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.578+0000 7fc0ee600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e806c530 con 0x7fc0e8151810 2024-03-27T06:14:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.578+0000 7fc0ee600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e8073150 con 0x7fc0e8158360 2024-03-27T06:14:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.578+0000 7fc0ee600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e806c7d0 con 0x7fc0e8151bf0 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0dfe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc0e8151bf0 0x7fc0e8072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0e7400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fc0e8151810 0x7fc0e80c1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33104/0 (socket says 172.21.15.132:33104) 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0e7400640 1 -- 172.21.15.132:0/238411425 learned_addr learned my addr 172.21.15.132:0/238411425 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0e7e00640 1 --1- 172.21.15.132:0/238411425 >> v1:172.21.15.134:6789/0 conn(0x7fc0e8158360 0x7fc0e8072a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37858/0 (socket says 172.21.15.132:37858) 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0e6a00640 1 -- 172.21.15.132:0/238411425 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170374015 0 0) 0x7fc0e8073150 con 0x7fc0e8158360 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0dfe00640 1 -- 172.21.15.132:0/238411425 >> v1:172.21.15.134:6789/0 conn(0x7fc0e8158360 legacy=0x7fc0e8072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0dfe00640 1 -- 172.21.15.132:0/238411425 >> v1:172.21.15.132:6789/0 conn(0x7fc0e8151810 legacy=0x7fc0e80c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.579+0000 7fc0dfe00640 1 -- 172.21.15.132:0/238411425 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e806cab0 con 0x7fc0e8151bf0 2024-03-27T06:14:17.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.580+0000 7fc0dfe00640 1 --2- 172.21.15.132:0/238411425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8151bf0 0x7fc0e8072500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00097a0 tx=0x7fc0d002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a54dc2fc935fb9a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.580+0000 7fc0e6a00640 1 -- 172.21.15.132:0/238411425 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0038070 con 0x7fc0e8151bf0 2024-03-27T06:14:17.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0e6a00640 1 -- 172.21.15.132:0/238411425 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d0031700 con 0x7fc0e8151bf0 2024-03-27T06:14:17.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0e6a00640 1 -- 172.21.15.132:0/238411425 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0031ad0 con 0x7fc0e8151bf0 2024-03-27T06:14:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 -- 172.21.15.132:0/238411425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8151bf0 msgr2=0x7fc0e8072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 --2- 172.21.15.132:0/238411425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8151bf0 0x7fc0e8072500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00097a0 tx=0x7fc0d002b820 comp rx=0 tx=0).stop 2024-03-27T06:14:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 -- 172.21.15.132:0/238411425 shutdown_connections 2024-03-27T06:14:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 --2- 172.21.15.132:0/238411425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8151bf0 0x7fc0e8072500 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 -- 172.21.15.132:0/238411425 >> 172.21.15.132:0/238411425 conn(0x7fc0e80c89a0 msgr2=0x7fc0e80c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 -- 172.21.15.132:0/238411425 shutdown_connections 2024-03-27T06:14:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.581+0000 7fc0ee600640 1 -- 172.21.15.132:0/238411425 wait complete. 2024-03-27T06:14:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.583+0000 7fc0ee600640 1 Processor -- start 2024-03-27T06:14:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.584+0000 7fc0ee600640 1 -- start start 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 0x7fc0e81f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 0x7fc0e81f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 0x7fc0e8153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8157e40 con 0x7fc0e8151810 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8157b60 con 0x7fc0e8158360 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0ee600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e80c86d0 con 0x7fc0e81535c0 2024-03-27T06:14:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 0x7fc0e81f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 0x7fc0e81f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0e7e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 0x7fc0e8153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 0x7fc0e81f02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37608/0 (socket says 172.21.15.132:37608) 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0e7400640 1 -- 172.21.15.132:0/2012355831 learned_addr learned my addr 172.21.15.132:0/2012355831 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 -- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 msgr2=0x7fc0e8153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 0x7fc0e8153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 -- 172.21.15.132:0/2012355831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 msgr2=0x7fc0e81f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 0x7fc0e81f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0dfe00640 1 -- 172.21.15.132:0/2012355831 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e8153f90 con 0x7fc0e8158360 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.585+0000 7fc0e7e00640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 0x7fc0e8153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0dfe00640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 0x7fc0e81f0820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0033000 tx=0x7fc0d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0043070 con 0x7fc0e8158360 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d00339b0 con 0x7fc0e8158360 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0038040 con 0x7fc0e8158360 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e81f0ef0 con 0x7fc0e8158360 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.586+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e80c7150 con 0x7fc0e8158360 2024-03-27T06:14:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.587+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e8068950 con 0x7fc0e8158360 2024-03-27T06:14:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.588+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc0d0046070 con 0x7fc0e8158360 2024-03-27T06:14:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.589+0000 7fc0df400640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 0x7fc0c406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.589+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc0d00ad500 con 0x7fc0e8158360 2024-03-27T06:14:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.589+0000 7fc0e7400640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 0x7fc0c406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:17.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.589+0000 7fc0e7400640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 0x7fc0c406ea20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8000f60 tx=0x7fc0d8007ec0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:17.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.593+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc0d0030070 con 0x7fc0e8158360 2024-03-27T06:14:17.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.792+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0e8154e10 con 0x7fc0e8158360 2024-03-27T06:14:17.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.793+0000 7fc0df400640 1 -- 172.21.15.132:0/2012355831 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fc0d007b550 con 0x7fc0e8158360 2024-03-27T06:14:17.784 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 msgr2=0x7fc0c406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 0x7fc0c406ea20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8000f60 tx=0x7fc0d8007ec0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 msgr2=0x7fc0e81f0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 0x7fc0e81f0820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0033000 tx=0x7fc0d00046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 shutdown_connections 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc0e81535c0 0x7fc0e8153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc0c406c560 0x7fc0c406ea20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0e8158360 0x7fc0e81f0820 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 --2- 172.21.15.132:0/2012355831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0e8151810 0x7fc0e81f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 >> 172.21.15.132:0/2012355831 conn(0x7fc0e80c89a0 msgr2=0x7fc0e80ca2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:17.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 shutdown_connections 2024-03-27T06:14:17.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:17.796+0000 7fc0ee600640 1 -- 172.21.15.132:0/2012355831 wait complete. 2024-03-27T06:14:17.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:14:18.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.027+0000 7f6705e00640 1 Processor -- start 2024-03-27T06:14:18.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.027+0000 7f6705e00640 1 -- start start 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f6705e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6700154f80 0x7f670015b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f6705e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f67000cb2f0 0x7f67000d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f6705e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f670015bb40 con 0x7f67000d0740 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f6705e00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f6700155360 con 0x7f6700154f80 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f6705e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f67000ccd30 con 0x7f67000cb2f0 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66f6000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f67000cb2f0 0x7f67000d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66fea00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6700154f80 0x7f670015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66ff400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f67000d0740 0x7f67000c8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37870/0 (socket says 172.21.15.132:37870) 2024-03-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66ff400640 1 -- 172.21.15.132:0/2799773748 learned_addr learned my addr 172.21.15.132:0/2799773748 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66fea00640 1 -- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f67000cb2f0 msgr2=0x7f67000d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66fea00640 1 --2- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f67000cb2f0 0x7f67000d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66fea00640 1 -- 172.21.15.132:0/2799773748 >> v1:172.21.15.134:6789/0 conn(0x7f67000d0740 legacy=0x7f67000c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.030+0000 7f66fea00640 1 -- 172.21.15.132:0/2799773748 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67000ccfd0 con 0x7f6700154f80 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.029+0000 7f66fe000640 1 -- 172.21.15.132:0/2799773748 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270675707 0 0) 0x7f670015bb40 con 0x7f67000d0740 2024-03-27T06:14:18.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f66fea00640 1 --2- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6700154f80 0x7f670015b600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097a0 tx=0x7f66f002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec5640f2f42ee3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f66fe000640 1 -- 172.21.15.132:0/2799773748 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00315a0 con 0x7f6700154f80 2024-03-27T06:14:18.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f66fe000640 1 -- 172.21.15.132:0/2799773748 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f0037070 con 0x7f6700154f80 2024-03-27T06:14:18.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f66fe000640 1 -- 172.21.15.132:0/2799773748 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0031960 con 0x7f6700154f80 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 -- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6700154f80 msgr2=0x7f670015b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 --2- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6700154f80 0x7f670015b600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097a0 tx=0x7f66f002b840 comp rx=0 tx=0).stop 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 -- 172.21.15.132:0/2799773748 shutdown_connections 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 --2- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f67000cb2f0 0x7f67000d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 --2- 172.21.15.132:0/2799773748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6700154f80 0x7f670015b600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.031+0000 7f6705e00640 1 -- 172.21.15.132:0/2799773748 >> 172.21.15.132:0/2799773748 conn(0x7f670006d9f0 msgr2=0x7f670006de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.032+0000 7f6705e00640 1 -- 172.21.15.132:0/2799773748 shutdown_connections 2024-03-27T06:14:18.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.032+0000 7f6705e00640 1 -- 172.21.15.132:0/2799773748 wait complete. 2024-03-27T06:14:18.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.033+0000 7f6705e00640 1 Processor -- start 2024-03-27T06:14:18.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.033+0000 7f6705e00640 1 -- start start 2024-03-27T06:14:18.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67000cb2f0 0x7f6700151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6700154f80 0x7f67000c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f67000cd870 con 0x7f67000cb2f0 2024-03-27T06:14:18.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f67000cd590 con 0x7f67000d0740 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f6705e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f67000cc0a0 con 0x7f6700154f80 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f66f6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f66ff400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6700154f80 0x7f67000c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f66fea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67000cb2f0 0x7f6700151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f66f6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56498/0 (socket says 172.21.15.132:56498) 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.034+0000 7f66f6000640 1 -- 172.21.15.132:0/4049924791 learned_addr learned my addr 172.21.15.132:0/4049924791 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 -- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6700154f80 msgr2=0x7f67000c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6700154f80 0x7f67000c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 -- 172.21.15.132:0/4049924791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67000cb2f0 msgr2=0x7f6700151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67000cb2f0 0x7f6700151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 -- 172.21.15.132:0/4049924791 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67001f3640 con 0x7f67000d0740 2024-03-27T06:14:18.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f6000640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66e800e9b0 tx=0x7f66e800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8017070 con 0x7f67000d0740 2024-03-27T06:14:18.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e800c9a0 con 0x7f67000d0740 2024-03-27T06:14:18.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e800cd90 con 0x7f67000d0740 2024-03-27T06:14:18.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670015d370 con 0x7f67000d0740 2024-03-27T06:14:18.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.035+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67001f3bc0 con 0x7f67000d0740 2024-03-27T06:14:18.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.037+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67000d0f60 con 0x7f67000d0740 2024-03-27T06:14:18.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.037+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f66e801a070 con 0x7f67000d0740 2024-03-27T06:14:18.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.038+0000 7f66f7e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 0x7f66dc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.038+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f66e808b6b0 con 0x7f67000d0740 2024-03-27T06:14:18.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.038+0000 7f66fea00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 0x7f66dc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.039+0000 7f66fea00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 0x7f66dc06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f66f0036000 tx=0x7f66f003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.042+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f67000d0f60 con 0x7f67000d0740 2024-03-27T06:14:18.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.243+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f67000cbe80 con 0x7f67000d0740 2024-03-27T06:14:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.243+0000 7f66f7e00640 1 -- 172.21.15.132:0/4049924791 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f66e80595e0 con 0x7f67000d0740 2024-03-27T06:14:18.234 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:14:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 msgr2=0x7f66dc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 0x7f66dc06e9f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f66f0036000 tx=0x7f66f003d040 comp rx=0 tx=0).stop 2024-03-27T06:14:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 msgr2=0x7f67000c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66e800e9b0 tx=0x7f66e800ee70 comp rx=0 tx=0).stop 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 shutdown_connections 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6700154f80 0x7f67000c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f66dc06c530 0x7f66dc06e9f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67000d0740 0x7f67000c5a50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 --2- 172.21.15.132:0/4049924791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67000cb2f0 0x7f6700151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 >> 172.21.15.132:0/4049924791 conn(0x7f670006d9f0 msgr2=0x7f67000c51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.246+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 shutdown_connections 2024-03-27T06:14:18.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.247+0000 7f6705e00640 1 -- 172.21.15.132:0/4049924791 wait complete. 2024-03-27T06:14:18.256 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:14:18.459 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.469+0000 7f6f23e00640 1 Processor -- start 2024-03-27T06:14:18.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.469+0000 7f6f23e00640 1 -- start start 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.470+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f24072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.470+0000 7f6f23e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f6f240c68a0 0x7f6f240c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.470+0000 7f6f23e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f24071780 con 0x7f6f24152790 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.470+0000 7f6f23e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f6f240c63f0 con 0x7f6f240c68a0 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.470+0000 7f6f23e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6f240c8520 con 0x7f6f24071f20 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f22a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6f24152790 0x7f6f24159020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37882/0 (socket says 172.21.15.132:37882) 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f22a00640 1 -- 172.21.15.132:0/2394590131 learned_addr learned my addr 172.21.15.132:0/2394590131 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:18.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f1a000640 1 --2- 172.21.15.132:0/2394590131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f24072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f23400640 1 --2- 172.21.15.132:0/2394590131 >> v2:172.21.15.132:3300/0 conn(0x7f6f240c68a0 0x7f6f240c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f22000640 1 -- 172.21.15.132:0/2394590131 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373138381 0 0) 0x7f6f14001110 con 0x7f6f24152790 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f22000640 1 -- 172.21.15.132:0/2394590131 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f08003520 con 0x7f6f24152790 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f1a000640 1 -- 172.21.15.132:0/2394590131 >> v2:172.21.15.132:3300/0 conn(0x7f6f240c68a0 msgr2=0x7f6f240c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f1a000640 1 --2- 172.21.15.132:0/2394590131 >> v2:172.21.15.132:3300/0 conn(0x7f6f240c68a0 0x7f6f240c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f1a000640 1 -- 172.21.15.132:0/2394590131 >> v1:172.21.15.134:6789/0 conn(0x7f6f24152790 legacy=0x7f6f24159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.471+0000 7f6f1a000640 1 -- 172.21.15.132:0/2394590131 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f24070ff0 con 0x7f6f24071f20 2024-03-27T06:14:18.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.472+0000 7f6f1a000640 1 --2- 172.21.15.132:0/2394590131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f24072340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0097a0 tx=0x7f6f0c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116df2294cb39007 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.472+0000 7f6f22000640 1 -- 172.21.15.132:0/2394590131 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c034070 con 0x7f6f24071f20 2024-03-27T06:14:18.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.472+0000 7f6f22000640 1 -- 172.21.15.132:0/2394590131 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0c030610 con 0x7f6f24071f20 2024-03-27T06:14:18.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.472+0000 7f6f22000640 1 -- 172.21.15.132:0/2394590131 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0309e0 con 0x7f6f24071f20 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 -- 172.21.15.132:0/2394590131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 msgr2=0x7f6f24072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 --2- 172.21.15.132:0/2394590131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f24072340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0097a0 tx=0x7f6f0c02b960 comp rx=0 tx=0).stop 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 -- 172.21.15.132:0/2394590131 shutdown_connections 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 --2- 172.21.15.132:0/2394590131 >> v2:172.21.15.132:3300/0 conn(0x7f6f240c68a0 0x7f6f240c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 --2- 172.21.15.132:0/2394590131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f24072340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 -- 172.21.15.132:0/2394590131 >> 172.21.15.132:0/2394590131 conn(0x7f6f240c89a0 msgr2=0x7f6f240c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 -- 172.21.15.132:0/2394590131 shutdown_connections 2024-03-27T06:14:18.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.473+0000 7f6f23e00640 1 -- 172.21.15.132:0/2394590131 wait complete. 2024-03-27T06:14:18.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.475+0000 7f6f23e00640 1 Processor -- start 2024-03-27T06:14:18.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.476+0000 7f6f23e00640 1 -- start start 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 0x7f6f24153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f24152790 0x7f6f24154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f24072880 con 0x7f6f24152790 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6f24071a00 con 0x7f6f240c68a0 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6f24152b70 con 0x7f6f24071f20 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f1a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 0x7f6f24153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35550/0 (socket says 172.21.15.132:35550) 2024-03-27T06:14:18.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 -- 172.21.15.132:0/365210880 learned_addr learned my addr 172.21.15.132:0/365210880 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f1a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 0x7f6f24153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56504/0 (socket says 172.21.15.132:56504) 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f23400640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f24152790 0x7f6f24154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 -- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 msgr2=0x7f6f24153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 0x7f6f24153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 -- 172.21.15.132:0/365210880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f24152790 msgr2=0x7f6f24154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f24152790 0x7f6f24154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 -- 172.21.15.132:0/365210880 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f241f3030 con 0x7f6f24071f20 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.477+0000 7f6f22a00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c3d0 tx=0x7f6f1400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.478+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14017070 con 0x7f6f24071f20 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.478+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f14005ce0 con 0x7f6f24071f20 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.478+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f241f3310 con 0x7f6f24071f20 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.478+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14007310 con 0x7f6f24071f20 2024-03-27T06:14:18.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.478+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f241f4350 con 0x7f6f24071f20 2024-03-27T06:14:18.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.479+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f240686c0 con 0x7f6f24071f20 2024-03-27T06:14:18.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.480+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6f1401a070 con 0x7f6f24071f20 2024-03-27T06:14:18.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.480+0000 7f6f1be00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 0x7f6f0006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.481+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6f1408aae0 con 0x7f6f24071f20 2024-03-27T06:14:18.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.481+0000 7f6f1a000640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 0x7f6f0006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.481+0000 7f6f1a000640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 0x7f6f0006e9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.485+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f6f24071780 con 0x7f6f24071f20 2024-03-27T06:14:18.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.698+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f240c5480 con 0x7f6f24071f20 2024-03-27T06:14:18.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.698+0000 7f6f1be00640 1 -- 172.21.15.132:0/365210880 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f6f14058a10 con 0x7f6f24071f20 2024-03-27T06:14:18.689 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.701+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 msgr2=0x7f6f0006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.701+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 0x7f6f0006e9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c03b040 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 msgr2=0x7f6f241f3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c3d0 tx=0x7f6f1400c890 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 shutdown_connections 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f24152790 0x7f6f24154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6f0006c530 0x7f6f0006e9f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f240c68a0 0x7f6f24153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 --2- 172.21.15.132:0/365210880 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6f24071f20 0x7f6f241f3850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 >> 172.21.15.132:0/365210880 conn(0x7f6f240c89a0 msgr2=0x7f6f240c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 shutdown_connections 2024-03-27T06:14:18.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.702+0000 7f6f23e00640 1 -- 172.21.15.132:0/365210880 wait complete. 2024-03-27T06:14:18.712 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:18.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.925+0000 7f268f200640 1 Processor -- start 2024-03-27T06:14:18.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.925+0000 7f268f200640 1 -- start start 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.926+0000 7f268f200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2690151830 0x7f26901574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.926+0000 7f268f200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.926+0000 7f268f200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2690157ad0 con 0x7f26900b1750 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.926+0000 7f268f200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2690071290 con 0x7f2690151830 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.926+0000 7f268f200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2690157db0 con 0x7f26900c64d0 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2690151830 0x7f26901574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2690151830 0x7f26901574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56510/0 (socket says 172.21.15.132:56510) 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35558/0 (socket says 172.21.15.132:35558) 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 -- 172.21.15.132:0/415023225 learned_addr learned my addr 172.21.15.132:0/415023225 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:18.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 -- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 msgr2=0x7f26900c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:18.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 -- 172.21.15.132:0/415023225 >> v1:172.21.15.132:6789/0 conn(0x7f26900b1750 legacy=0x7f26900730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268de00640 1 -- 172.21.15.132:0/415023225 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2690158090 con 0x7f2690151830 2024-03-27T06:14:18.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268ca00640 1 -- 172.21.15.132:0/415023225 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761293132 0 0) 0x7f2690157ad0 con 0x7f26900b1750 2024-03-27T06:14:18.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.927+0000 7f268d400640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.928+0000 7f268de00640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2690151830 0x7f26901574f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e50 tx=0x7f268002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1ed755b71b482a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.928+0000 7f268ca00640 1 -- 172.21.15.132:0/415023225 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268002cb90 con 0x7f2690151830 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.928+0000 7f268ca00640 1 -- 172.21.15.132:0/415023225 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f268002cd30 con 0x7f2690151830 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.928+0000 7f268ca00640 1 -- 172.21.15.132:0/415023225 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680034610 con 0x7f2690151830 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 -- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2690151830 msgr2=0x7f26901574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2690151830 0x7f26901574f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e50 tx=0x7f268002bff0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 -- 172.21.15.132:0/415023225 shutdown_connections 2024-03-27T06:14:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26900c64d0 0x7f26900c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 --2- 172.21.15.132:0/415023225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2690151830 0x7f26901574f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 -- 172.21.15.132:0/415023225 >> 172.21.15.132:0/415023225 conn(0x7f26900cc7a0 msgr2=0x7f26900c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 -- 172.21.15.132:0/415023225 shutdown_connections 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.929+0000 7f268f200640 1 -- 172.21.15.132:0/415023225 wait complete. 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.931+0000 7f268f200640 1 Processor -- start 2024-03-27T06:14:18.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.931+0000 7f268f200640 1 -- start start 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 0x7f2690150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 0x7f2690151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2690072b90 con 0x7f26900c64d0 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f26900728b0 con 0x7f26900b1750 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268f200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2690151c10 con 0x7f2690151830 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268e800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 0x7f2690151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56524/0 (socket says 172.21.15.132:56524) 2024-03-27T06:14:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268e800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 0x7f2690151e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35570/0 (socket says 172.21.15.132:35570) 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 -- 172.21.15.132:0/1100310250 learned_addr learned my addr 172.21.15.132:0/1100310250 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268d400640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 0x7f2690150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 -- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 msgr2=0x7f2690151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 0x7f2690151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 -- 172.21.15.132:0/1100310250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 msgr2=0x7f2690150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 0x7f2690150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.932+0000 7f268de00640 1 -- 172.21.15.132:0/1100310250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26900c0800 con 0x7f26900b1750 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f268d400640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 0x7f2690150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f268de00640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2680000f80 tx=0x7f2680004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680043020 con 0x7f26900b1750 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f268003e450 con 0x7f26900b1750 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680009510 con 0x7f26900b1750 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26901ef0f0 con 0x7f26900b1750 2024-03-27T06:14:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.933+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26901ee330 con 0x7f26900b1750 2024-03-27T06:14:18.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.934+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26900686c0 con 0x7f26900b1750 2024-03-27T06:14:18.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.935+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2680005d10 con 0x7f26900b1750 2024-03-27T06:14:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.936+0000 7f2686000640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 0x7f266c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.936+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f26800adff0 con 0x7f26900b1750 2024-03-27T06:14:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.936+0000 7f268d400640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 0x7f266c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.937+0000 7f268d400640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 0x7f266c06e9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f26900bf5a0 tx=0x7f2678008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:18.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:18.941+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f268007c5e0 con 0x7f26900b1750 2024-03-27T06:14:19.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.179+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2690070eb0 con 0x7f26900b1750 2024-03-27T06:14:19.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.179+0000 7f2686000640 1 -- 172.21.15.132:0/1100310250 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6877 (secure 0 0 0) 0x7f268007bee0 con 0x7f26900b1750 2024-03-27T06:14:19.170 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:19.170 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:13.590498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:14:19.171 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 11 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 msgr2=0x7f266c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 0x7f266c06e9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f26900bf5a0 tx=0x7f2678008040 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 msgr2=0x7f2690153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2680000f80 tx=0x7f2680004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 shutdown_connections 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2690151830 0x7f2690151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26900c64d0 0x7f2690150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f266c06c530 0x7f266c06e9f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 --2- 172.21.15.132:0/1100310250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26900b1750 0x7f2690153450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 >> 172.21.15.132:0/1100310250 conn(0x7f26900cc7a0 msgr2=0x7f2690072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:19.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 shutdown_connections 2024-03-27T06:14:19.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.182+0000 7f268f200640 1 -- 172.21.15.132:0/1100310250 wait complete. 2024-03-27T06:14:19.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:19.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.407+0000 7f3c40e00640 1 Processor -- start 2024-03-27T06:14:19.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.407+0000 7f3c40e00640 1 -- start start 2024-03-27T06:14:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.408+0000 7f3c40e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3c3c06ce00 0x7f3c3c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.408+0000 7f3c40e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3c3c15ad60 0x7f3c3c15b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c40e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c3c15ebe0 con 0x7f3c3c1548b0 2024-03-27T06:14:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c40e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f3c3c0cbc80 con 0x7f3c3c15ad60 2024-03-27T06:14:19.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c40e00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3c3c15e060 con 0x7f3c3c06ce00 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3b400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3c3c1548b0 0x7f3c3c0cc930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37886/0 (socket says 172.21.15.132:37886) 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3b400640 1 -- 172.21.15.132:0/148415293 learned_addr learned my addr 172.21.15.132:0/148415293 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3be00640 1 --2- 172.21.15.132:0/148415293 >> v2:172.21.15.132:3300/0 conn(0x7f3c3c15ad60 0x7f3c3c15b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c32a00640 1 --2- 172.21.15.132:0/148415293 >> v2:172.21.15.134:3300/0 conn(0x7f3c3c06ce00 0x7f3c3c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3aa00640 1 -- 172.21.15.132:0/148415293 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092544481 0 0) 0x7f3c3c15ebe0 con 0x7f3c3c1548b0 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3aa00640 1 -- 172.21.15.132:0/148415293 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c20003520 con 0x7f3c3c1548b0 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3be00640 1 -- 172.21.15.132:0/148415293 >> v2:172.21.15.134:3300/0 conn(0x7f3c3c06ce00 msgr2=0x7f3c3c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3be00640 1 --2- 172.21.15.132:0/148415293 >> v2:172.21.15.134:3300/0 conn(0x7f3c3c06ce00 0x7f3c3c0cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3be00640 1 -- 172.21.15.132:0/148415293 >> v1:172.21.15.134:6789/0 conn(0x7f3c3c1548b0 legacy=0x7f3c3c0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.409+0000 7f3c3be00640 1 -- 172.21.15.132:0/148415293 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3c15e340 con 0x7f3c3c15ad60 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.410+0000 7f3c3be00640 1 --2- 172.21.15.132:0/148415293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c15b190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0cc8b0 tx=0x7f3c3402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37a67c7c480c13a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c3aa00640 1 -- 172.21.15.132:0/148415293 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34039070 con 0x7f3c3c15ad60 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c3aa00640 1 -- 172.21.15.132:0/148415293 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3402cd00 con 0x7f3c3c15ad60 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c3aa00640 1 -- 172.21.15.132:0/148415293 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34034660 con 0x7f3c3c15ad60 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c40e00640 1 -- 172.21.15.132:0/148415293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 msgr2=0x7f3c3c15b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c40e00640 1 --2- 172.21.15.132:0/148415293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c15b190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0cc8b0 tx=0x7f3c3402bf50 comp rx=0 tx=0).stop 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c40e00640 1 -- 172.21.15.132:0/148415293 shutdown_connections 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c40e00640 1 --2- 172.21.15.132:0/148415293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c15b190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.411+0000 7f3c40e00640 1 --2- 172.21.15.132:0/148415293 >> v2:172.21.15.134:3300/0 conn(0x7f3c3c06ce00 0x7f3c3c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.412+0000 7f3c40e00640 1 -- 172.21.15.132:0/148415293 >> 172.21.15.132:0/148415293 conn(0x7f3c3c06e190 msgr2=0x7f3c3c067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.412+0000 7f3c40e00640 1 -- 172.21.15.132:0/148415293 shutdown_connections 2024-03-27T06:14:19.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.412+0000 7f3c40e00640 1 -- 172.21.15.132:0/148415293 wait complete. 2024-03-27T06:14:19.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.413+0000 7f3c40e00640 1 Processor -- start 2024-03-27T06:14:19.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.413+0000 7f3c40e00640 1 -- start start 2024-03-27T06:14:19.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c3c06ce00 0x7f3c3c0cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3c3c1548b0 0x7f3c3c149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c15a840 con 0x7f3c3c15ad60 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c15e900 con 0x7f3c3c06ce00 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.414+0000 7f3c40e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3c3c15dc90 con 0x7f3c3c1548b0 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c3c06ce00 0x7f3c3c0cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c32a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3c3c1548b0 0x7f3c3c149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37650/0 (socket says 172.21.15.132:37650) 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 -- 172.21.15.132:0/3714893394 learned_addr learned my addr 172.21.15.132:0/3714893394 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 -- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3c3c1548b0 msgr2=0x7f3c3c149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3c3c1548b0 0x7f3c3c149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 -- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c3c06ce00 msgr2=0x7f3c3c0cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c3c06ce00 0x7f3c3c0cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 -- 172.21.15.132:0/3714893394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3c1f0250 con 0x7f3c3c15ad60 2024-03-27T06:14:19.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c3be00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c3402c560 tx=0x7f3c34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.415+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34039050 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.416+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c34045070 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.416+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34009450 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.416+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3c1f0530 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.416+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3c1f0af0 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.417+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c3c1f0cf0 con 0x7f3c3c15ad60 2024-03-27T06:14:19.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.418+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3c3403e850 con 0x7f3c3c15ad60 2024-03-27T06:14:19.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.419+0000 7f3c33e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 0x7f3c1806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.419+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c340ae300 con 0x7f3c3c15ad60 2024-03-27T06:14:19.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.419+0000 7f3c3b400640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 0x7f3c1806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.419+0000 7f3c3b400640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 0x7f3c1806e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c005880 tx=0x7f3c2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.424+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3c3c1f0cf0 con 0x7f3c3c15ad60 2024-03-27T06:14:19.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.630+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c3c067400 con 0x7f3c3c15ad60 2024-03-27T06:14:19.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.634+0000 7f3c33e00640 1 -- 172.21.15.132:0/3714893394 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f3c3407c3e0 con 0x7f3c3c15ad60 2024-03-27T06:14:19.625 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:19.625 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.636+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 msgr2=0x7f3c1806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.636+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 0x7f3c1806e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c005880 tx=0x7f3c2c009040 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 msgr2=0x7f3c3c1590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c3402c560 tx=0x7f3c34004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 shutdown_connections 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c3c15ad60 0x7f3c3c1590b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3c3c1548b0 0x7f3c3c149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3c1806c530 0x7f3c1806e9f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 --2- 172.21.15.132:0/3714893394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c3c06ce00 0x7f3c3c0cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 >> 172.21.15.132:0/3714893394 conn(0x7f3c3c06e190 msgr2=0x7f3c3c152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 shutdown_connections 2024-03-27T06:14:19.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.637+0000 7f3c40e00640 1 -- 172.21.15.132:0/3714893394 wait complete. 2024-03-27T06:14:19.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:19.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.860+0000 7f8c9fe00640 1 Processor -- start 2024-03-27T06:14:19.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.860+0000 7f8c9fe00640 1 -- start start 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 0x7f8c98159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9fe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8c9806bb70 0x7f8c98159c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9fe00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c980722d0 con 0x7f8c9806d870 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9fe00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f8c980720a0 con 0x7f8c9806bb70 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9fe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8c980731c0 con 0x7f8c9806b790 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 0x7f8c98159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8c9806bb70 0x7f8c98159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8c9806bb70 0x7f8c98159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56552/0 (socket says 172.21.15.132:56552) 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 0x7f8c98159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35596/0 (socket says 172.21.15.132:35596) 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 -- 172.21.15.132:0/3489969727 learned_addr learned my addr 172.21.15.132:0/3489969727 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 -- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 msgr2=0x7f8c98159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 --2- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 0x7f8c98159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 -- 172.21.15.132:0/3489969727 >> v1:172.21.15.132:6789/0 conn(0x7f8c9806d870 legacy=0x7f8c980c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.862+0000 7f8c9ca00640 1 -- 172.21.15.132:0/3489969727 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c980725b0 con 0x7f8c9806bb70 2024-03-27T06:14:19.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c9ca00640 1 --2- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806bb70 0x7f8c98159c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009870 tx=0x7f8c8402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaa6715e00229b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c97e00640 1 -- 172.21.15.132:0/3489969727 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402f830 con 0x7f8c9806bb70 2024-03-27T06:14:19.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c97e00640 1 -- 172.21.15.132:0/3489969727 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8402f9d0 con 0x7f8c9806bb70 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c97e00640 1 -- 172.21.15.132:0/3489969727 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402fda0 con 0x7f8c9806bb70 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c9fe00640 1 -- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806bb70 msgr2=0x7f8c98159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.864+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806bb70 0x7f8c98159c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009870 tx=0x7f8c8402ba10 comp rx=0 tx=0).stop 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 -- 172.21.15.132:0/3489969727 shutdown_connections 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806bb70 0x7f8c98159c90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/3489969727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806b790 0x7f8c98159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 -- 172.21.15.132:0/3489969727 >> 172.21.15.132:0/3489969727 conn(0x7f8c9806fc20 msgr2=0x7f8c98070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 -- 172.21.15.132:0/3489969727 shutdown_connections 2024-03-27T06:14:19.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.865+0000 7f8c9fe00640 1 -- 172.21.15.132:0/3489969727 wait complete. 2024-03-27T06:14:19.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.866+0000 7f8c9fe00640 1 Processor -- start 2024-03-27T06:14:19.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.866+0000 7f8c9fe00640 1 -- start start 2024-03-27T06:14:19.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806bb70 0x7f8c981f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c9806d870 0x7f8c981f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8c98072e50 con 0x7f8c9806d870 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8c98072b70 con 0x7f8c9806b790 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.867+0000 7f8c9fe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9806c1d0 con 0x7f8c9806bb70 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56566/0 (socket says 172.21.15.132:56566) 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 -- 172.21.15.132:0/4146924067 learned_addr learned my addr 172.21.15.132:0/4146924067 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9ca00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806bb70 0x7f8c981f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9de00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c9806d870 0x7f8c981f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 -- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806bb70 msgr2=0x7f8c981f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806bb70 0x7f8c981f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 -- 172.21.15.132:0/4146924067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c9806d870 msgr2=0x7f8c981f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c9806d870 0x7f8c981f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 -- 172.21.15.132:0/4146924067 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c981f38f0 con 0x7f8c9806b790 2024-03-27T06:14:19.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9d400640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00e870 tx=0x7f8c8c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c017070 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8c00c940 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c980cbc40 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.868+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00cd30 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.869+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9815bee0 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.870+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c98068840 con 0x7f8c9806b790 2024-03-27T06:14:19.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.870+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8c8c01a070 con 0x7f8c9806b790 2024-03-27T06:14:19.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.871+0000 7f8c95600640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:19.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.871+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8c8c08b560 con 0x7f8c9806b790 2024-03-27T06:14:19.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.872+0000 7f8c9ca00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:19.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.872+0000 7f8c9ca00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 0x7f8c7806e9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c84036000 tx=0x7f8c84008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:19.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:19.876+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f8c8c059530 con 0x7f8c9806b790 2024-03-27T06:14:20.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.101+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c980c76b0 con 0x7f8c9806b790 2024-03-27T06:14:20.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.102+0000 7f8c95600640 1 -- 172.21.15.132:0/4146924067 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7f8c8c05d830 con 0x7f8c9806b790 2024-03-27T06:14:20.093 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:20.093 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:14:20.094 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:14:20.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 msgr2=0x7f8c7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 0x7f8c7806e9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c84036000 tx=0x7f8c84008030 comp rx=0 tx=0).stop 2024-03-27T06:14:20.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 msgr2=0x7f8c9815c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00e870 tx=0x7f8c8c00ed30 comp rx=0 tx=0).stop 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 shutdown_connections 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c9806d870 0x7f8c981f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8c9806bb70 0x7f8c981f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8c7806c530 0x7f8c7806e9f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 --2- 172.21.15.132:0/4146924067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c9806b790 0x7f8c9815c830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 >> 172.21.15.132:0/4146924067 conn(0x7f8c9806fc20 msgr2=0x7f8c98158690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 shutdown_connections 2024-03-27T06:14:20.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.105+0000 7f8c9fe00640 1 -- 172.21.15.132:0/4146924067 wait complete. 2024-03-27T06:14:20.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-27T06:14:02.985226+0000', 'modified': '2024-03-27T06:14:19.619957+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4273, 'mds_1': 4270, 'mds_2': 4244}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'c', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.132:6837/690619582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6835', 'nonce': 690619582}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 690619582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.134:6838/2702381292', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6836', 'nonce': 2702381292}, {'type': 'v1', 'addr': '172.21.15.134:6838', 'nonce': 2702381292}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4273': {'gid': 4273, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.134:6839/3802035585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6837', 'nonce': 3802035585}, {'type': 'v1', 'addr': '172.21.15.134:6839', 'nonce': 3802035585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4273, 'qdb_cluster': [4244, 4270, 4273]} 2024-03-27T06:14:20.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-27T06:14:20.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-27T06:14:20.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.944+0000 7f491ea00640 1 Processor -- start 2024-03-27T06:14:20.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.945+0000 7f491ea00640 1 -- start start 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.946+0000 7f491ea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f491806b520 0x7f491806b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.946+0000 7f491ea00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f491815b790 con 0x7f491815aa00 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.946+0000 7f491ea00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49180c12d0 con 0x7f49180b11e0 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.946+0000 7f491ea00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f49180c1080 con 0x7f491806b520 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f49180b11e0 0x7f49180c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37898/0 (socket says 172.21.15.132:37898) 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917e00640 1 -- 172.21.15.132:0/1557089118 learned_addr learned my addr 172.21.15.132:0/1557089118 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f490f400640 1 --2- 172.21.15.132:0/1557089118 >> v2:172.21.15.132:3300/0 conn(0x7f491806b520 0x7f491806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994794728 0 0) 0x7f49180c12d0 con 0x7f49180b11e0 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48fc003520 con 0x7f49180b11e0 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980580855 0 0) 0x7f491815b790 con 0x7f491815aa00 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49180c12d0 con 0x7f491815aa00 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f490f400640 1 -- 172.21.15.132:0/1557089118 >> v1:172.21.15.134:6789/0 conn(0x7f49180b11e0 legacy=0x7f49180c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1476244755 0 0) 0x7f48fc003520 con 0x7f49180b11e0 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f490f400640 1 -- 172.21.15.132:0/1557089118 >> v1:172.21.15.132:6789/0 conn(0x7f491815aa00 legacy=0x7f491815ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.947+0000 7f490f400640 1 -- 172.21.15.132:0/1557089118 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49180c15b0 con 0x7f491806b520 2024-03-27T06:14:20.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.948+0000 7f490f400640 1 --2- 172.21.15.132:0/1557089118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f491806b960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4900009720 tx=0x7f490002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d3f879076d7f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:20.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.948+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908001670 con 0x7f491806b520 2024-03-27T06:14:20.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.948+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900037070 con 0x7f491806b520 2024-03-27T06:14:20.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f4917400640 1 -- 172.21.15.132:0/1557089118 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000317e0 con 0x7f491806b520 2024-03-27T06:14:20.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 -- 172.21.15.132:0/1557089118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 msgr2=0x7f491806b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 --2- 172.21.15.132:0/1557089118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f491806b960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4900009720 tx=0x7f490002b860 comp rx=0 tx=0).stop 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 -- 172.21.15.132:0/1557089118 shutdown_connections 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 --2- 172.21.15.132:0/1557089118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f491806b960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 -- 172.21.15.132:0/1557089118 >> 172.21.15.132:0/1557089118 conn(0x7f49180cc060 msgr2=0x7f49180cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 -- 172.21.15.132:0/1557089118 shutdown_connections 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.949+0000 7f491ea00640 1 -- 172.21.15.132:0/1557089118 wait complete. 2024-03-27T06:14:20.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.951+0000 7f491ea00640 1 Processor -- start 2024-03-27T06:14:20.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.951+0000 7f491ea00640 1 -- start start 2024-03-27T06:14:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f4918154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 0x7f4918156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f49180b15c0 con 0x7f491806b520 2024-03-27T06:14:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49180c1c00 con 0x7f49180b11e0 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ea00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f49180c2090 con 0x7f491815aa00 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f4917e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f4918154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56574/0 (socket says 172.21.15.132:56574) 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f491ca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 0x7f4918156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.952+0000 7f490f400640 1 -- 172.21.15.132:0/2965914521 learned_addr learned my addr 172.21.15.132:0/2965914521 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 -- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 msgr2=0x7f4918156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 0x7f4918156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 -- 172.21.15.132:0/2965914521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 msgr2=0x7f4918154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f4918154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 -- 172.21.15.132:0/2965914521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49181f39b0 con 0x7f49180b11e0 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f491ca00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 0x7f4918156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f4917e00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f4918154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f490f400640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4900009840 tx=0x7f49000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f4914c00640 1 -- 172.21.15.132:0/2965914521 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490004a070 con 0x7f49180b11e0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f4914c00640 1 -- 172.21.15.132:0/2965914521 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900031e80 con 0x7f49180b11e0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49181f3c90 con 0x7f49180b11e0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f4914c00640 1 -- 172.21.15.132:0/2965914521 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900032670 con 0x7f49180b11e0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.953+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49181f4250 con 0x7f49180b11e0 2024-03-27T06:14:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.956+0000 7f4914c00640 1 -- 172.21.15.132:0/2965914521 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f490004d070 con 0x7f49180b11e0 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.957+0000 7f4914c00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 0x7f48f406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.957+0000 7f4914c00640 1 -- 172.21.15.132:0/2965914521 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49000acff0 con 0x7f49180b11e0 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.957+0000 7f4917e00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 0x7f48f406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4921628740 1 Processor -- start 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4917e00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 0x7f48f406ea20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4908001500 tx=0x7f4908001490 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:20.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4921628740 1 -- start start 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4921628740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225bc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.958+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 0x5611225fb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4921628740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56112257d920 con 0x561122605850 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5611224cdfa0 con 0x5611225ba430 2024-03-27T06:14:20.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x56112251ff90 con 0x5611225bb9d0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35622/0 (socket says 172.21.15.132:35622) 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 -- 172.21.15.132:0/3448776976 learned_addr learned my addr 172.21.15.132:0/3448776976 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f490f400640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225bc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f491ca00640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 0x5611225fb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 -- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 msgr2=0x5611225fb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 0x5611225fb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 -- 172.21.15.132:0/3448776976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 msgr2=0x5611225bc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225bc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 -- 172.21.15.132:0/3448776976 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611222342d0 con 0x5611225bb9d0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f491ca00640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 0x5611225fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f490f400640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225bc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.959+0000 7f4917e00640 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4908004ea0 tx=0x7f4908013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83eceb325f108b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:20.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4907e00640 1 -- 172.21.15.132:0/3448776976 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908026070 con 0x5611225bb9d0 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4907e00640 1 -- 172.21.15.132:0/3448776976 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4908012350 con 0x5611225bb9d0 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4907e00640 1 -- 172.21.15.132:0/3448776976 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908012700 con 0x5611225bb9d0 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 -- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 msgr2=0x5611225bbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4908004ea0 tx=0x7f4908013000 comp rx=0 tx=0).stop 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 -- 172.21.15.132:0/3448776976 shutdown_connections 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225ba430 0x5611225fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225bc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 --2- 172.21.15.132:0/3448776976 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225bbdb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 -- 172.21.15.132:0/3448776976 >> 172.21.15.132:0/3448776976 conn(0x5611225b8140 msgr2=0x5611225baa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 -- 172.21.15.132:0/3448776976 shutdown_connections 2024-03-27T06:14:20.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.960+0000 7f4921628740 1 -- 172.21.15.132:0/3448776976 wait complete. 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.961+0000 7f4921628740 1 -- start start 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.961+0000 7f4921628740 1 Processor -- start 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x5611226031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56112251bf40 con 0x561122602460 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5611224c8650 con 0x5611225bb9d0 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x5611224dba10 con 0x561122605850 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4921628740 10 client.0 _ll_register_callbacks cb 0x56112251aff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35626/0 (socket says 172.21.15.132:35626) 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f490f400640 1 -- 172.21.15.132:0/1755855296 learned_addr learned my addr 172.21.15.132:0/1755855296 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.962+0000 7f491ca00640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x5611226031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 -- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 msgr2=0x5611225f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 -- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 msgr2=0x5611226031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x5611226031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 -- 172.21.15.132:0/1755855296 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5611222fe6b0 con 0x561122605850 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4917e00640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f490f400640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f490002b8a0 tx=0x7f490007b100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0dfd8880c84e185 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4904c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3301/0 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900079380 con 0x561122605850 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mon.2 v2:172.21.15.134:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4900093120 con 0x561122605850 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4908012350 con 0x561122605850 2024-03-27T06:14:20.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f4900076d10 con 0x561122605850 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4921628740 10 client.4502 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x561122602010 con 0x561122605850 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.963+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mon.2 v2:172.21.15.134:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4908012350 con 0x561122605850 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mon.2 v2:172.21.15.134:3301/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7f4900076eb0 con 0x561122605850 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 10 client.4502 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 10 client.4502 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 20 client.4502 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x56112260c820 0x56112261b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 4 client.4502 mds_command: new command op to 4244 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 --> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56112252d4d0 con 0x56112260c820 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4917e00640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x56112260c820 0x56112261b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4917e00640 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x56112260c820 0x56112261b650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc6d3babdd2018c8 server_cookie=e71a0c83a1f2a7b3 in_seq=0 out_seq=0 2024-03-27T06:14:20.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:20.964+0000 7f4904c00640 10 client.4502 ms_handle_connect on v2:172.21.15.132:6835/690619582 2024-03-27T06:14:21.382 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:14:21.391+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-27T06:14:24.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4904c00640 1 -- 172.21.15.132:0/1755855296 <== mds.2 v2:172.21.15.132:6835/690619582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56112252d4d0 con 0x56112260c820 2024-03-27T06:14:24.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4904c00640 10 client.4502 handle_command_reply: tid=0 2024-03-27T06:14:24.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 client.4502 shutdown 2024-03-27T06:14:24.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 msgr2=0x561122610520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f490002b8a0 tx=0x7f490007b100 comp rx=0 tx=0).stop 2024-03-27T06:14:24.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 shutdown_connections 2024-03-27T06:14:24.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x56112260c820 0x56112261b650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x5611226031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122605850 0x561122610520 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 --2- 172.21.15.132:0/1755855296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4904c00640 0 client.4502 ms_handle_reset on v2:172.21.15.132:6835/690619582 2024-03-27T06:14:24.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.633+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 >> 172.21.15.132:0/1755855296 conn(0x5611225ba430 msgr2=0x5611225bbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.634+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 shutdown_connections 2024-03-27T06:14:24.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.634+0000 7f4921628740 1 -- 172.21.15.132:0/1755855296 wait complete. 2024-03-27T06:14:24.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.634+0000 7f4921628740 20 client.4502 trim_cache size 0 max 16384 2024-03-27T06:14:24.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.665+0000 7f4921628740 1 Processor -- start 2024-03-27T06:14:24.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.665+0000 7f4921628740 1 -- start start 2024-03-27T06:14:24.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225b7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4918150980 con 0x561122605850 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56112251bf40 con 0x5611225bb9d0 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x5611224dba10 con 0x561122602460 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.666+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57414/0 (socket says 172.21.15.132:57414) 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 -- 172.21.15.132:0/4015511631 learned_addr learned my addr 172.21.15.132:0/4015511631 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f491ca00640 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 -- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 msgr2=0x5611225b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 -- 172.21.15.132:0/4015511631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 msgr2=0x5611225f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 -- 172.21.15.132:0/4015511631 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112260fd20 con 0x561122602460 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f4917e00640 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.667+0000 7f490f400640 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4900009810 tx=0x7f4900046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c9fa2f136bc851 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.668+0000 7f4907400640 1 -- 172.21.15.132:0/4015511631 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900079380 con 0x561122602460 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.668+0000 7f4907400640 1 -- 172.21.15.132:0/4015511631 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900037040 con 0x561122602460 2024-03-27T06:14:24.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.668+0000 7f4907400640 1 -- 172.21.15.132:0/4015511631 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000ab300 con 0x561122602460 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 -- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 msgr2=0x5611225f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4900009810 tx=0x7f4900046000 comp rx=0 tx=0).stop 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 -- 172.21.15.132:0/4015511631 shutdown_connections 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122605850 0x5611225f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x561122602460 0x5611225f76a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 --2- 172.21.15.132:0/4015511631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5611225bb9d0 0x5611225b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.660 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 -- 172.21.15.132:0/4015511631 >> 172.21.15.132:0/4015511631 conn(0x5611225ba430 msgr2=0x5611225b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 -- 172.21.15.132:0/4015511631 shutdown_connections 2024-03-27T06:14:24.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.669+0000 7f4921628740 1 -- 172.21.15.132:0/4015511631 wait complete. 2024-03-27T06:14:24.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.670+0000 7f4921628740 1 -- start start 2024-03-27T06:14:24.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.670+0000 7f4921628740 1 Processor -- start 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x561122620d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56112260c6a0 con 0x561122602460 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561122602a80 con 0x561122605850 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x56112260cd30 con 0x5611225bb9d0 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4921628740 10 client.0 _ll_register_callbacks cb 0x7f490007ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f491ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57422/0 (socket says 172.21.15.132:57422) 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 -- 172.21.15.132:0/828023614 learned_addr learned my addr 172.21.15.132:0/828023614 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f491ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51360/0 (socket says 172.21.15.132:51360) 2024-03-27T06:14:24.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f490f400640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x561122620d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 -- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 msgr2=0x5611225ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 -- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 msgr2=0x561122620d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x561122620d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.672+0000 7f4917e00640 1 -- 172.21.15.132:0/828023614 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5611226774f0 con 0x5611225bb9d0 2024-03-27T06:14:24.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f491ca00640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4917e00640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f490801f000 tx=0x7f4908012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77dc35eed542a747 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4904c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3301/0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490803c060 con 0x5611225bb9d0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mon.2 v2:172.21.15.134:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4908001bf0 con 0x5611225bb9d0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4900037040 con 0x5611225bb9d0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f4908042050 con 0x5611225bb9d0 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4921628740 10 client.4508 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:24.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.673+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x5611225f9e40 con 0x5611225bb9d0 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mon.2 v2:172.21.15.134:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900037040 con 0x5611225bb9d0 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mon.2 v2:172.21.15.134:3301/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7f4908045020 con 0x5611225bb9d0 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 10 client.4508 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 10 client.4508 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 20 client.4508 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x561122609ad0 0x561122682eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 4 client.4508 mds_command: new command op to 4244 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-27T06:14:24.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 --> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] -- command(tid 0: {"prefix": "status"}) -- 0x561122527820 con 0x561122609ad0 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.674+0000 7f490f400640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x561122609ad0 0x561122682eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f490f400640 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x561122609ad0 0x561122682eb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=707346cefe90206d server_cookie=aa0ed4d4bf34fe8d in_seq=0 out_seq=0 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4904c00640 10 client.4508 ms_handle_connect on v2:172.21.15.132:6835/690619582 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4904c00640 1 -- 172.21.15.132:0/828023614 <== mds.2 v2:172.21.15.132:6835/690619582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x561122527820 con 0x561122609ad0 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4904c00640 10 client.4508 handle_command_reply: tid=0 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 client.4508 shutdown 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 msgr2=0x5611225b9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f490801f000 tx=0x7f4908012000 comp rx=0 tx=0).stop 2024-03-27T06:14:24.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 shutdown_connections 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:6835/690619582,v1:172.21.15.132:6837/690619582] conn(0x561122609ad0 0x561122682eb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561122605850 0x5611225ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561122602460 0x561122620d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 --2- 172.21.15.132:0/828023614 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x5611225bb9d0 0x5611225b9c10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4904c00640 0 client.4508 ms_handle_reset on v2:172.21.15.132:6835/690619582 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 >> 172.21.15.132:0/828023614 conn(0x5611225ba430 msgr2=0x5611225b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.675+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 shutdown_connections 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f4921628740 1 -- 172.21.15.132:0/828023614 wait complete. 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f4921628740 20 client.4508 trim_cache size 0 max 16384 2024-03-27T06:14:24.670 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_fsid": "3dfa4fc8-5726-4b88-a1bf-7da524e3644e", 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "whoami": 2, 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "id": 4244, 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "want_state": "up:active", 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "up:active", 2024-03-27T06:14:24.671 INFO:teuthology.orchestra.run.smithi132.stdout: "fs_name": "cephfs", 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "rank_uptime": 11.081885097000001, 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "mdsmap_epoch": 12, 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch": 21, 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch_barrier": 21, 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout: "uptime": 29.239788662999999 2024-03-27T06:14:24.672 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:14:24.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 msgr2=0x7f48f406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 0x7f48f406ea20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4908001500 tx=0x7f4908001490 comp rx=0 tx=0).stop 2024-03-27T06:14:24.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 msgr2=0x7f49180b0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4900009840 tx=0x7f49000046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 shutdown_connections 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f491815aa00 0x7f4918156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f48f406c560 0x7f48f406ea20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49180b11e0 0x7f49180b0110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.676+0000 7f491ea00640 1 --2- 172.21.15.132:0/2965914521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f491806b520 0x7f4918154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.677+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 >> 172.21.15.132:0/2965914521 conn(0x7f49180cc060 msgr2=0x7f49180ca720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.677+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 shutdown_connections 2024-03-27T06:14:24.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.677+0000 7f491ea00640 1 -- 172.21.15.132:0/2965914521 wait complete. 2024-03-27T06:14:24.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-27T06:14:24.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.898+0000 7fc374000640 1 Processor -- start 2024-03-27T06:14:24.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.898+0000 7fc374000640 1 -- start start 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.900+0000 7fc374000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 0x7fc36c0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.900+0000 7fc374000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc36c06fb10 0x7fc36c0ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.900+0000 7fc374000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36c158670 con 0x7fc36c158290 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.900+0000 7fc374000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fc36c06ed40 con 0x7fc36c06fb10 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.900+0000 7fc374000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c15b2f0 con 0x7fc36c0c9c80 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 0x7fc36c0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc371600640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fc36c158290 0x7fc36c15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42212/0 (socket says 172.21.15.132:42212) 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc371600640 1 -- 172.21.15.132:0/3989420198 learned_addr learned my addr 172.21.15.132:0/3989420198 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc372000640 1 --2- 172.21.15.132:0/3989420198 >> v2:172.21.15.134:3300/0 conn(0x7fc36c06fb10 0x7fc36c0ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc370c00640 1 -- 172.21.15.132:0/3989420198 >> v2:172.21.15.134:3300/0 conn(0x7fc36c06fb10 msgr2=0x7fc36c0ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc370c00640 1 --2- 172.21.15.132:0/3989420198 >> v2:172.21.15.134:3300/0 conn(0x7fc36c06fb10 0x7fc36c0ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc370c00640 1 -- 172.21.15.132:0/3989420198 >> v1:172.21.15.132:6789/0 conn(0x7fc36c158290 legacy=0x7fc36c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.901+0000 7fc370c00640 1 -- 172.21.15.132:0/3989420198 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c06e2f0 con 0x7fc36c0c9c80 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.902+0000 7fc370c00640 1 --2- 172.21.15.132:0/3989420198 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 0x7fc36c0ce290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3580097a0 tx=0x7fc35802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ce998f524ddc7c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.902+0000 7fc36be00640 1 -- 172.21.15.132:0/3989420198 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580315a0 con 0x7fc36c0c9c80 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.902+0000 7fc36be00640 1 -- 172.21.15.132:0/3989420198 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc358037070 con 0x7fc36c0c9c80 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.902+0000 7fc36be00640 1 -- 172.21.15.132:0/3989420198 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358031960 con 0x7fc36c0c9c80 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 -- 172.21.15.132:0/3989420198 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 msgr2=0x7fc36c0ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 --2- 172.21.15.132:0/3989420198 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 0x7fc36c0ce290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3580097a0 tx=0x7fc35802b840 comp rx=0 tx=0).stop 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 -- 172.21.15.132:0/3989420198 shutdown_connections 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 --2- 172.21.15.132:0/3989420198 >> v2:172.21.15.134:3300/0 conn(0x7fc36c06fb10 0x7fc36c0ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 --2- 172.21.15.132:0/3989420198 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c0c9c80 0x7fc36c0ce290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 -- 172.21.15.132:0/3989420198 >> 172.21.15.132:0/3989420198 conn(0x7fc36c067280 msgr2=0x7fc36c067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 -- 172.21.15.132:0/3989420198 shutdown_connections 2024-03-27T06:14:24.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.903+0000 7fc374000640 1 -- 172.21.15.132:0/3989420198 wait complete. 2024-03-27T06:14:24.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.905+0000 7fc374000640 1 Processor -- start 2024-03-27T06:14:24.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.905+0000 7fc374000640 1 -- start start 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc36c0c9c80 0x7fc36c152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc36c158290 0x7fc36c1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c06f690 con 0x7fc36c0c9c80 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c06c9b0 con 0x7fc36c158290 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc374000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c06e5d0 con 0x7fc36c06fb10 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57430/0 (socket says 172.21.15.132:57430) 2024-03-27T06:14:24.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 -- 172.21.15.132:0/2225481707 learned_addr learned my addr 172.21.15.132:0/2225481707 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc370c00640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc36c0c9c80 0x7fc36c152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc372000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc36c158290 0x7fc36c1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 -- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc36c158290 msgr2=0x7fc36c1f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc36c158290 0x7fc36c1f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 -- 172.21.15.132:0/2225481707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc36c0c9c80 msgr2=0x7fc36c152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc36c0c9c80 0x7fc36c152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.906+0000 7fc371600640 1 -- 172.21.15.132:0/2225481707 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c14bf60 con 0x7fc36c06fb10 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc371600640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c3f0 tx=0x7fc36000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc369600640 1 -- 172.21.15.132:0/2225481707 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360017070 con 0x7fc36c06fb10 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc369600640 1 -- 172.21.15.132:0/2225481707 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc360005ce0 con 0x7fc36c06fb10 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc369600640 1 -- 172.21.15.132:0/2225481707 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3600072f0 con 0x7fc36c06fb10 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c15a200 con 0x7fc36c06fb10 2024-03-27T06:14:24.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.907+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c15a7a0 con 0x7fc36c06fb10 2024-03-27T06:14:24.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.909+0000 7fc369600640 1 -- 172.21.15.132:0/2225481707 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc36001a070 con 0x7fc36c06fb10 2024-03-27T06:14:24.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.910+0000 7fc369600640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 0x7fc34c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.910+0000 7fc369600640 1 -- 172.21.15.132:0/2225481707 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc36008a9a0 con 0x7fc36c06fb10 2024-03-27T06:14:24.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.911+0000 7fc370c00640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 0x7fc34c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.911+0000 7fc376dd4740 1 Processor -- start 2024-03-27T06:14:24.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.911+0000 7fc370c00640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 0x7fc34c06e9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc358036000 tx=0x7fc35803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.911+0000 7fc376dd4740 1 -- start start 2024-03-27T06:14:24.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d2f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d2f0430 0x55d73d331230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d73d2b3920 con 0x55d73d2f0430 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d73d203fa0 con 0x55d73d33b850 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55d73d255f90 con 0x55d73d2f19d0 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d2f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc372000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d2f0430 0x55d73d331230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57434/0 (socket says 172.21.15.132:57434) 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d2f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51388/0 (socket says 172.21.15.132:51388) 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.912+0000 7fc371600640 1 -- 172.21.15.132:0/84066350 learned_addr learned my addr 172.21.15.132:0/84066350 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 -- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 msgr2=0x55d73d2f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d2f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 -- 172.21.15.132:0/84066350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d2f0430 msgr2=0x55d73d331230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d2f0430 0x55d73d331230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 -- 172.21.15.132:0/84066350 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d73cf6a2d0 con 0x55d73d2f19d0 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc371600640 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc360060f60 tx=0x7fc3600583f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64a1ebf6728f0839 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc35dc00640 1 -- 172.21.15.132:0/84066350 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360088f40 con 0x55d73d2f19d0 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc35dc00640 1 -- 172.21.15.132:0/84066350 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc360056a00 con 0x55d73d2f19d0 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.913+0000 7fc35dc00640 1 -- 172.21.15.132:0/84066350 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36005f070 con 0x55d73d2f19d0 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 msgr2=0x55d73d2f1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc360060f60 tx=0x7fc3600583f0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- 172.21.15.132:0/84066350 shutdown_connections 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d2f0430 0x55d73d331230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d2f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 --2- 172.21.15.132:0/84066350 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d2f1db0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- 172.21.15.132:0/84066350 >> 172.21.15.132:0/84066350 conn(0x55d73d2ee140 msgr2=0x55d73d2f0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- 172.21.15.132:0/84066350 shutdown_connections 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- 172.21.15.132:0/84066350 wait complete. 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.914+0000 7fc376dd4740 1 -- start start 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.915+0000 7fc376dd4740 1 Processor -- start 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d33b850 0x55d73d32cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d342ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d73d251f40 con 0x55d73d342a40 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d73d1fe650 con 0x55d73d2f19d0 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55d73d211a10 con 0x55d73d33b850 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc376dd4740 10 client.0 _ll_register_callbacks cb 0x55d73d250ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:24.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d33b850 0x55d73d32cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc372000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d342ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51398/0 (socket says 172.21.15.132:51398) 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 -- 172.21.15.132:0/1911335555 learned_addr learned my addr 172.21.15.132:0/1911335555 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 -- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d33b850 msgr2=0x55d73d32cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d33b850 0x55d73d32cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 -- 172.21.15.132:0/1911335555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 msgr2=0x55d73d342ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d342ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc371600640 1 -- 172.21.15.132:0/1911335555 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d73d0346b0 con 0x55d73d2f19d0 2024-03-27T06:14:24.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.916+0000 7fc372000640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d342ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc371600640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc360060f60 tx=0x7fc360000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b5133b9f2f38ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360075060 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc360059be0 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc360056a00 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7fc360059de0 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 10 client.4639 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55d73d32e840 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc360056a00 con 0x55d73d2f19d0 2024-03-27T06:14:24.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7fc360007490 con 0x55d73d2f19d0 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 10 client.4639 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 10 client.4639 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 20 client.4639 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d32d1e0 0x55d73d345cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 4 client.4639 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-27T06:14:24.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.917+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 --> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d73d2634d0 con 0x55d73d32d1e0 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.918+0000 7fc370c00640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d32d1e0 0x55d73d345cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.918+0000 7fc370c00640 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d32d1e0 0x55d73d345cd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e946bc9420f60033 server_cookie=ed067244d05a1c36 in_seq=0 out_seq=0 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.918+0000 7fc352000640 10 client.4639 ms_handle_connect on v2:172.21.15.134:6836/2702381292 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc352000640 1 -- 172.21.15.132:0/1911335555 <== mds.1 v2:172.21.15.134:6836/2702381292 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d73d2634d0 con 0x55d73d32d1e0 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc352000640 10 client.4639 handle_command_reply: tid=0 2024-03-27T06:14:24.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 client.4639 shutdown 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 msgr2=0x55d73d336a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc360060f60 tx=0x7fc360000bc0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 shutdown_connections 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d32d1e0 0x55d73d345cd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d342ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d33b850 0x55d73d32cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc352000640 0 client.4639 ms_handle_reset on v2:172.21.15.134:6836/2702381292 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 --2- 172.21.15.132:0/1911335555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d2f19d0 0x55d73d336a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 >> 172.21.15.132:0/1911335555 conn(0x55d73d2f0430 msgr2=0x55d73d2f1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 shutdown_connections 2024-03-27T06:14:24.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.919+0000 7fc376dd4740 1 -- 172.21.15.132:0/1911335555 wait complete. 2024-03-27T06:14:24.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.920+0000 7fc376dd4740 20 client.4639 trim_cache size 0 max 16384 2024-03-27T06:14:24.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.939+0000 7fc376dd4740 1 Processor -- start 2024-03-27T06:14:24.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.939+0000 7fc376dd4740 1 -- start start 2024-03-27T06:14:24.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d332cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d337a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c0b18f0 con 0x55d73d342a40 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d73d211a10 con 0x55d73d33b850 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55d73d1fe650 con 0x55d73d2f19d0 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d332cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51404/0 (socket says 172.21.15.132:51404) 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 -- 172.21.15.132:0/4158403652 learned_addr learned my addr 172.21.15.132:0/4158403652 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d332cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57442/0 (socket says 172.21.15.132:57442) 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc372000640 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d337a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 -- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 msgr2=0x55d73d332cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d332cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 -- 172.21.15.132:0/4158403652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 msgr2=0x55d73d337a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d337a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc370c00640 1 -- 172.21.15.132:0/4158403652 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d73d336df0 con 0x55d73d33b850 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.940+0000 7fc372000640 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d337a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc370c00640 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc35802be50 tx=0x7fc358030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a48b0632b45e5811 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc35d200640 1 -- 172.21.15.132:0/4158403652 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35805a070 con 0x55d73d33b850 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc35d200640 1 -- 172.21.15.132:0/4158403652 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc358037040 con 0x55d73d33b850 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc35d200640 1 -- 172.21.15.132:0/4158403652 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580550d0 con 0x55d73d33b850 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc376dd4740 1 -- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 msgr2=0x55d73d333230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.941+0000 7fc376dd4740 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc35802be50 tx=0x7fc358030000 comp rx=0 tx=0).stop 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 -- 172.21.15.132:0/4158403652 shutdown_connections 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d342a40 0x55d73d337a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d33b850 0x55d73d333230 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 --2- 172.21.15.132:0/4158403652 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d332cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 -- 172.21.15.132:0/4158403652 >> 172.21.15.132:0/4158403652 conn(0x55d73d2f0430 msgr2=0x55d73d32cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 -- 172.21.15.132:0/4158403652 shutdown_connections 2024-03-27T06:14:24.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 -- 172.21.15.132:0/4158403652 wait complete. 2024-03-27T06:14:24.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 -- start start 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.942+0000 7fc376dd4740 1 Processor -- start 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d33b850 0x55d73d3a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 0x55d73d3a6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d73d33c880 con 0x55d73d33b850 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d73d251f40 con 0x55d73d342a40 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55d73d337890 con 0x55d73d2f19d0 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.943+0000 7fc376dd4740 10 client.0 _ll_register_callbacks cb 0x7fc364009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc372000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 0x55d73d3a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57454/0 (socket says 172.21.15.132:57454) 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 -- 172.21.15.132:0/2141525621 learned_addr learned my addr 172.21.15.132:0/2141525621 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc370c00640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d33b850 0x55d73d3a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 -- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 msgr2=0x55d73d3a6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 0x55d73d3a6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 -- 172.21.15.132:0/2141525621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d33b850 msgr2=0x55d73d3a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d33b850 0x55d73d3a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 -- 172.21.15.132:0/2141525621 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55d73d330210 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc372000640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 0x55d73d3a6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc371600640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc360000fb0 tx=0x7fc360005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2248bd6fa1f77732 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc352000640 10 client.0 ms_handle_connect on v2:172.21.15.134:3301/0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.944+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360066060 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mon.2 v2:172.21.15.134:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc360059be0 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc358037040 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7fc36005cc60 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 10 client.4526 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55d73d330210 con 0x55d73d2f19d0 2024-03-27T06:14:24.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mon.2 v2:172.21.15.134:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc358037040 con 0x55d73d2f19d0 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mon.2 v2:172.21.15.134:3301/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7fc360044b90 con 0x55d73d2f19d0 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 10 client.4526 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 10 client.4526 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 20 client.4526 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d335f10 0x55d73d3362f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 4 client.4526 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 --> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] -- command(tid 0: {"prefix": "status"}) -- 0x55d73d25d820 con 0x55d73d335f10 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.945+0000 7fc370c00640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d335f10 0x55d73d3362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc370c00640 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d335f10 0x55d73d3362f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef357cab044551ef server_cookie=d59b6422c32c46b7 in_seq=0 out_seq=0 2024-03-27T06:14:24.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc352000640 10 client.4526 ms_handle_connect on v2:172.21.15.134:6836/2702381292 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc352000640 1 -- 172.21.15.132:0/2141525621 <== mds.1 v2:172.21.15.134:6836/2702381292 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d73d25d820 con 0x55d73d335f10 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc352000640 10 client.4526 handle_command_reply: tid=0 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 client.4526 shutdown 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 msgr2=0x55d73d337e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc360000fb0 tx=0x7fc360005f90 comp rx=0 tx=0).stop 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 shutdown_connections 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d73d342a40 0x55d73d3a6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:6836/2702381292,v1:172.21.15.134:6838/2702381292] conn(0x55d73d335f10 0x55d73d3362f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d73d33b850 0x55d73d3a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 --2- 172.21.15.132:0/2141525621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55d73d2f19d0 0x55d73d337e10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc352000640 0 client.4526 ms_handle_reset on v2:172.21.15.134:6836/2702381292 2024-03-27T06:14:24.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 >> 172.21.15.132:0/2141525621 conn(0x55d73d2f0430 msgr2=0x55d73d332cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.946+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 shutdown_connections 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc376dd4740 1 -- 172.21.15.132:0/2141525621 wait complete. 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc376dd4740 20 client.4526 trim_cache size 0 max 16384 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_fsid": "3dfa4fc8-5726-4b88-a1bf-7da524e3644e", 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "whoami": 1, 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "id": 4270, 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "want_state": "up:active", 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "up:active", 2024-03-27T06:14:24.944 INFO:teuthology.orchestra.run.smithi132.stdout: "fs_name": "cephfs", 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout: "rank_uptime": 16.383607108, 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout: "mdsmap_epoch": 12, 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch": 21, 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch_barrier": 21, 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout: "uptime": 29.569952533999999 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:14:24.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 msgr2=0x7fc34c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 0x7fc34c06e9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc358036000 tx=0x7fc35803d040 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 msgr2=0x7fc36c159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c3f0 tx=0x7fc36000c8b0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 shutdown_connections 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc34c06c530 0x7fc34c06e9f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc36c158290 0x7fc36c1f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc36c0c9c80 0x7fc36c152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 --2- 172.21.15.132:0/2225481707 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc36c06fb10 0x7fc36c159980 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 >> 172.21.15.132:0/2225481707 conn(0x7fc36c067280 msgr2=0x7fc36c0c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 shutdown_connections 2024-03-27T06:14:24.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:24.947+0000 7fc374000640 1 -- 172.21.15.132:0/2225481707 wait complete. 2024-03-27T06:14:24.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-27T06:14:25.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.160+0000 7f47d4e00640 1 Processor -- start 2024-03-27T06:14:25.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.160+0000 7f47d4e00640 1 -- start start 2024-03-27T06:14:25.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47d4e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d015b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47d4e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f47d015bf30 0x7f47d00c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47d4e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d006ba60 con 0x7f47d006c3d0 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47d4e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f47d015ba80 con 0x7f47d015bf30 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47d4e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f47d006bd40 con 0x7f47d0154e00 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47cde00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d015b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47c5400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f47d015bf30 0x7f47d00c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47ce800640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f47d006c3d0 0x7f47d00c7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42228/0 (socket says 172.21.15.132:42228) 2024-03-27T06:14:25.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47ce800640 1 -- 172.21.15.132:0/2701311550 learned_addr learned my addr 172.21.15.132:0/2701311550 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:25.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47cde00640 1 -- 172.21.15.132:0/2701311550 >> v2:172.21.15.132:3300/0 conn(0x7f47d015bf30 msgr2=0x7f47d00c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47cde00640 1 --2- 172.21.15.132:0/2701311550 >> v2:172.21.15.132:3300/0 conn(0x7f47d015bf30 0x7f47d00c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47cde00640 1 -- 172.21.15.132:0/2701311550 >> v1:172.21.15.132:6789/0 conn(0x7f47d006c3d0 legacy=0x7f47d00c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.162+0000 7f47cde00640 1 -- 172.21.15.132:0/2701311550 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d00c8230 con 0x7f47d0154e00 2024-03-27T06:14:25.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.163+0000 7f47cde00640 1 --2- 172.21.15.132:0/2701311550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d015b540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47c00097a0 tx=0x7f47c002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0e3f52a885d048b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47cd400640 1 -- 172.21.15.132:0/2701311550 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0035020 con 0x7f47d0154e00 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47cd400640 1 -- 172.21.15.132:0/2701311550 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c0031810 con 0x7f47d0154e00 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47cd400640 1 -- 172.21.15.132:0/2701311550 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0031be0 con 0x7f47d0154e00 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 -- 172.21.15.132:0/2701311550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 msgr2=0x7f47d015b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 --2- 172.21.15.132:0/2701311550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d015b540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47c00097a0 tx=0x7f47c002b8b0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 -- 172.21.15.132:0/2701311550 shutdown_connections 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 --2- 172.21.15.132:0/2701311550 >> v2:172.21.15.132:3300/0 conn(0x7f47d015bf30 0x7f47d00c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 --2- 172.21.15.132:0/2701311550 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d015b540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 -- 172.21.15.132:0/2701311550 >> 172.21.15.132:0/2701311550 conn(0x7f47d00d0150 msgr2=0x7f47d00d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 -- 172.21.15.132:0/2701311550 shutdown_connections 2024-03-27T06:14:25.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.164+0000 7f47d4e00640 1 -- 172.21.15.132:0/2701311550 wait complete. 2024-03-27T06:14:25.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.166+0000 7f47d4e00640 1 Processor -- start 2024-03-27T06:14:25.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.166+0000 7f47d4e00640 1 -- start start 2024-03-27T06:14:25.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d006c3d0 0x7f47d01553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 0x7f47d01f0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01551e0 con 0x7f47d015bf30 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f47d00c87f0 con 0x7f47d006c3d0 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47d4e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f47d00c9250 con 0x7f47d0154e00 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57472/0 (socket says 172.21.15.132:57472) 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47c5400640 1 -- 172.21.15.132:0/138498361 learned_addr learned my addr 172.21.15.132:0/138498361 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.167+0000 7f47cde00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d006c3d0 0x7f47d01553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47ce800640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 0x7f47d01f0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 -- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d006c3d0 msgr2=0x7f47d01553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d006c3d0 0x7f47d01553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 -- 172.21.15.132:0/138498361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 msgr2=0x7f47d01f0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 0x7f47d01f0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 -- 172.21.15.132:0/138498361 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d00cc160 con 0x7f47d0154e00 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47ce800640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 0x7f47d01f0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c5400640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e8b0 tx=0x7f47b800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c6a00640 1 -- 172.21.15.132:0/138498361 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8017070 con 0x7f47d0154e00 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c6a00640 1 -- 172.21.15.132:0/138498361 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b800c8e0 con 0x7f47d0154e00 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d00cc440 con 0x7f47d0154e00 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47c6a00640 1 -- 172.21.15.132:0/138498361 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b800ccb0 con 0x7f47d0154e00 2024-03-27T06:14:25.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.168+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d00cd2b0 con 0x7f47d0154e00 2024-03-27T06:14:25.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.170+0000 7f47c6a00640 1 -- 172.21.15.132:0/138498361 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f47b801a070 con 0x7f47d0154e00 2024-03-27T06:14:25.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.172+0000 7f47c6a00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 0x7f47ac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.172+0000 7f47c6a00640 1 -- 172.21.15.132:0/138498361 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f47b808b9f0 con 0x7f47d0154e00 2024-03-27T06:14:25.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.172+0000 7f47cde00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 0x7f47ac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.173+0000 7f47cde00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 0x7f47ac06e9f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47c0036000 tx=0x7f47c003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.173+0000 7f47d7b6f740 1 Processor -- start 2024-03-27T06:14:25.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.173+0000 7f47d7b6f740 1 -- start start 2024-03-27T06:14:25.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a899db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a898430 0x55e95a8d9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e95a85b920 con 0x55e95a8999d0 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e95a7abfa0 con 0x55e95a898430 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55e95a7fdf90 con 0x55e95a8e3850 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47cde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a899db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57488/0 (socket says 172.21.15.132:57488) 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 -- 172.21.15.132:0/3709403270 learned_addr learned my addr 172.21.15.132:0/3709403270 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47ce800640 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a898430 0x55e95a8d9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 -- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a898430 msgr2=0x55e95a8d9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a898430 0x55e95a8d9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 -- 172.21.15.132:0/3709403270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 msgr2=0x55e95a899db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a899db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.174+0000 7f47c5400640 1 -- 172.21.15.132:0/3709403270 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e95a5122d0 con 0x55e95a8e3850 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47c5400640 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47b8055d80 tx=0x7f47b8059aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4b1efc109227b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47bea00640 1 -- 172.21.15.132:0/3709403270 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8021070 con 0x55e95a8e3850 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47bea00640 1 -- 172.21.15.132:0/3709403270 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8071030 con 0x55e95a8e3850 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47bea00640 1 -- 172.21.15.132:0/3709403270 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8055f90 con 0x55e95a8e3850 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 msgr2=0x55e95a89a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47b8055d80 tx=0x7f47b8059aa0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3709403270 shutdown_connections 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a898430 0x55e95a8d9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a89a2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3709403270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a899db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.175+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3709403270 >> 172.21.15.132:0/3709403270 conn(0x55e95a896140 msgr2=0x55e95a898a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.176+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3709403270 shutdown_connections 2024-03-27T06:14:25.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.176+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3709403270 wait complete. 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.176+0000 7f47d7b6f740 1 -- start start 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.176+0000 7f47d7b6f740 1 Processor -- start 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.177+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.177+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.177+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8db830 0x55e95a8b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.177+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e95a7f9f40 con 0x55e95a8db830 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e95a7a6650 con 0x55e95a8e3850 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55e95a7b9a10 con 0x55e95a8999d0 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47cde00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47d7b6f740 10 client.0 _ll_register_callbacks cb 0x55e95a7f8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51448/0 (socket says 172.21.15.132:51448) 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47cde00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57502/0 (socket says 172.21.15.132:57502) 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 -- 172.21.15.132:0/3582005489 learned_addr learned my addr 172.21.15.132:0/3582005489 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47ce800640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8db830 0x55e95a8b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 -- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 msgr2=0x55e95a8b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 -- 172.21.15.132:0/3582005489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8db830 msgr2=0x55e95a8b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8db830 0x55e95a8b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 -- 172.21.15.132:0/3582005489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e95a5dc6b0 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47cde00640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47c5400640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f47b80595d0 tx=0x7f47b8060050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7c007e9c197e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47b7400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.178+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8071e60 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f47b805aba0 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47b8071030 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f47b805adc0 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 10 client.4654 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55e95a8b7330 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8071030 con 0x55e95a8e3850 2024-03-27T06:14:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7f47b8058d70 con 0x55e95a8e3850 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 10 client.4654 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 10 client.4654 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 20 client.4654 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f7190 0x55e95a8f9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 4 client.4654 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 --> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e95a80b4d0 con 0x55e95a8f7190 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.179+0000 7f47cde00640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f7190 0x55e95a8f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.180+0000 7f47cde00640 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f7190 0x55e95a8f9580 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7a9d4787ed3ddea server_cookie=9068e62fec1f798c in_seq=0 out_seq=0 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.180+0000 7f47b7400640 10 client.4654 ms_handle_connect on v2:172.21.15.134:6837/3802035585 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47b7400640 1 -- 172.21.15.132:0/3582005489 <== mds.0 v2:172.21.15.134:6837/3802035585 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e95a80b4d0 con 0x55e95a8f7190 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47b7400640 10 client.4654 handle_command_reply: tid=0 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 client.4654 shutdown 2024-03-27T06:14:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 msgr2=0x55e95a8b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f47b80595d0 tx=0x7f47b8060050 comp rx=0 tx=0).stop 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 shutdown_connections 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f7190 0x55e95a8f9580 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8db830 0x55e95a8b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8b4c80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3582005489 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8999d0 0x55e95a8b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47b7400640 0 client.4654 ms_handle_reset on v2:172.21.15.134:6837/3802035585 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 >> 172.21.15.132:0/3582005489 conn(0x55e95a898430 msgr2=0x55e95a899db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.181+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 shutdown_connections 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.182+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3582005489 wait complete. 2024-03-27T06:14:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.182+0000 7f47d7b6f740 20 client.4654 trim_cache size 0 max 16384 2024-03-27T06:14:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.201+0000 7f47d7b6f740 1 Processor -- start 2024-03-27T06:14:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.201+0000 7f47d7b6f740 1 -- start start 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a8e2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01f4390 con 0x55e95a8999d0 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e95a7f9f40 con 0x55e95a8e3850 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55e95a7a6650 con 0x55e95a8db830 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47cde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a8e2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57512/0 (socket says 172.21.15.132:57512) 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 -- 172.21.15.132:0/758934856 learned_addr learned my addr 172.21.15.132:0/758934856 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47ce800640 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 -- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 msgr2=0x55e95a8991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 -- 172.21.15.132:0/758934856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 msgr2=0x55e95a8e2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a8e2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.202+0000 7f47c5400640 1 -- 172.21.15.132:0/758934856 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e95a8c2a70 con 0x55e95a8db830 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47c5400640 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47b8059860 tx=0x7f47b8085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=841681c832ed2ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47be000640 1 -- 172.21.15.132:0/758934856 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8071e60 con 0x55e95a8db830 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47be000640 1 -- 172.21.15.132:0/758934856 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8071040 con 0x55e95a8db830 2024-03-27T06:14:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47be000640 1 -- 172.21.15.132:0/758934856 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8062450 con 0x55e95a8db830 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 -- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 msgr2=0x55e95a898c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47b8059860 tx=0x7f47b8085040 comp rx=0 tx=0).stop 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 -- 172.21.15.132:0/758934856 shutdown_connections 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8e3850 0x55e95a8991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8db830 0x55e95a898c90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/758934856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a8e2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.203+0000 7f47d7b6f740 1 -- 172.21.15.132:0/758934856 >> 172.21.15.132:0/758934856 conn(0x55e95a898430 msgr2=0x55e95a88f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.204+0000 7f47d7b6f740 1 -- 172.21.15.132:0/758934856 shutdown_connections 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.204+0000 7f47d7b6f740 1 -- 172.21.15.132:0/758934856 wait complete. 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.204+0000 7f47d7b6f740 1 -- start start 2024-03-27T06:14:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.204+0000 7f47d7b6f740 1 Processor -- start 2024-03-27T06:14:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8db830 0x55e95a8fec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a8b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e95a8d4c90 con 0x55e95a8999d0 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e95a8de720 con 0x55e95a8db830 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x55e95a8f5310 con 0x55e95a8e3850 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47d7b6f740 10 client.0 _ll_register_callbacks cb 0x7f47c0009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47cde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.205+0000 7f47c5400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8db830 0x55e95a8fec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a8b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47cde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38320/0 (socket says 172.21.15.132:38320) 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47cde00640 1 -- 172.21.15.132:0/3516108973 learned_addr learned my addr 172.21.15.132:0/3516108973 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 -- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8db830 msgr2=0x55e95a8fec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8db830 0x55e95a8fec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 -- 172.21.15.132:0/3516108973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 msgr2=0x55e95a94ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 -- 172.21.15.132:0/3516108973 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55e95a898f70 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47cde00640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47ce800640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a8b1560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55e95a8dd850 tx=0x7f47c8015780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=488b1cada7e463a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47b7400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3301/0 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c80070d0 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mon.2 v2:172.21.15.134:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f47c8007270 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.206+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47b8071040 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f47c8007470 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 10 client.4544 fetch_fsmap learned FSMap version 12 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55e95a89a3f0 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mon.2 v2:172.21.15.134:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8071040 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mon.2 v2:172.21.15.134:3301/0 5 ==== fsmap(e 12) ==== 4744+0+0 (secure 0 0 0) 0x7f47c802b080 con 0x55e95a8e3850 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 10 client.4544 fetch_fsmap finished waiting for FSMap version 12 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 10 client.4544 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 20 client.4544 populate_metadata read hostname 'smithi132' 2024-03-27T06:14:25.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f4680 0x55e95a961000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 4 client.4544 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 --> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] -- command(tid 0: {"prefix": "status"}) -- 0x55e95a805820 con 0x55e95a8f4680 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.207+0000 7f47cde00640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f4680 0x55e95a961000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47cde00640 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f4680 0x55e95a961000 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416603119b10213d server_cookie=ef842142593a6c8b in_seq=0 out_seq=0 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47b7400640 10 client.4544 ms_handle_connect on v2:172.21.15.134:6837/3802035585 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47b7400640 1 -- 172.21.15.132:0/3516108973 <== mds.0 v2:172.21.15.134:6837/3802035585 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55e95a805820 con 0x55e95a8f4680 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47b7400640 10 client.4544 handle_command_reply: tid=0 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 client.4544 shutdown 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 msgr2=0x55e95a8b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a8b1560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55e95a8dd850 tx=0x7f47c8015780 comp rx=0 tx=0).stop 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 shutdown_connections 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:6837/3802035585,v1:172.21.15.134:6839/3802035585] conn(0x55e95a8f4680 0x55e95a961000 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x55e95a8e3850 0x55e95a8b1560 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e95a8db830 0x55e95a8fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47b7400640 0 client.4544 ms_handle_reset on v2:172.21.15.134:6837/3802035585 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 --2- 172.21.15.132:0/3516108973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e95a8999d0 0x55e95a94ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 >> 172.21.15.132:0/3516108973 conn(0x55e95a898430 msgr2=0x55e95a88f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 shutdown_connections 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.208+0000 7f47d7b6f740 1 -- 172.21.15.132:0/3516108973 wait complete. 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d7b6f740 20 client.4544 trim_cache size 0 max 16384 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_fsid": "3dfa4fc8-5726-4b88-a1bf-7da524e3644e", 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stdout: "whoami": 0, 2024-03-27T06:14:25.205 INFO:teuthology.orchestra.run.smithi132.stdout: "id": 4273, 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "want_state": "up:active", 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "up:active", 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "fs_name": "cephfs", 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "rank_uptime": 21.700932011999999, 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "mdsmap_epoch": 12, 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch": 21, 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_epoch_barrier": 21, 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout: "uptime": 29.831685663999998 2024-03-27T06:14:25.206 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 msgr2=0x7f47ac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 0x7f47ac06e9f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47c0036000 tx=0x7f47c003c040 comp rx=0 tx=0).stop 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 msgr2=0x7f47d014d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e8b0 tx=0x7f47b800ed70 comp rx=0 tx=0).stop 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 shutdown_connections 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d015bf30 0x7f47d01f0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f47d0154e00 0x7f47d014d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f47ac06c530 0x7f47ac06e9f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 --2- 172.21.15.132:0/138498361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d006c3d0 0x7f47d01553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 >> 172.21.15.132:0/138498361 conn(0x7f47d00d0150 msgr2=0x7f47d00d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 shutdown_connections 2024-03-27T06:14:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.209+0000 7f47d4e00640 1 -- 172.21.15.132:0/138498361 wait complete. 2024-03-27T06:14:25.218 INFO:teuthology.run_tasks:Running task sequential... 2024-03-27T06:14:25.228 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-27T06:14:25.229 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-27T06:14:25.229 INFO:teuthology.task.print:Enabling mgr modules 2024-03-27T06:14:25.229 INFO:teuthology.run_tasks:Running task kclient... 2024-03-27T06:14:25.253 INFO:tasks.kclient:Mounting kernel clients... 2024-03-27T06:14:25.253 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-27T06:14:25.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:25.253 DEBUG:teuthology.orchestra.run.smithi132:> ip netns list 2024-03-27T06:14:25.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:25.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link delete ceph-brx 2024-03-27T06:14:25.378 INFO:teuthology.orchestra.run.smithi132.stderr:Cannot find device "ceph-brx" 2024-03-27T06:14:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:14:25.382 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-27T06:14:25.382 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-27T06:14:25.382 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:25.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-27T06:14:25.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.588+0000 7f4665a00640 1 Processor -- start 2024-03-27T06:14:25.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.588+0000 7f4665a00640 1 -- start start 2024-03-27T06:14:25.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.589+0000 7f4665a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 0x7f466015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.589+0000 7f4665a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.589+0000 7f4665a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4660158670 con 0x7f46600c9c80 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f4665a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f466006ed40 con 0x7f4660158290 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f4665a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f466006e2f0 con 0x7f466015b120 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465ea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 0x7f466015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465f400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f46600c9c80 0x7f466015e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42234/0 (socket says 172.21.15.132:42234) 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57532/0 (socket says 172.21.15.132:57532) 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465ea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 0x7f466015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38324/0 (socket says 172.21.15.132:38324) 2024-03-27T06:14:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465f400640 1 -- 172.21.15.132:0/2332005876 learned_addr learned my addr 172.21.15.132:0/2332005876 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 -- 172.21.15.132:0/2332005876 >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 msgr2=0x7f466015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 --2- 172.21.15.132:0/2332005876 >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 0x7f466015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 -- 172.21.15.132:0/2332005876 >> v1:172.21.15.132:6789/0 conn(0x7f46600c9c80 legacy=0x7f466015e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.590+0000 7f465e000640 1 -- 172.21.15.132:0/2332005876 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466006c6d0 con 0x7f466015b120 2024-03-27T06:14:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.591+0000 7f465e000640 1 --2- 172.21.15.132:0/2332005876 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f464c0097a0 tx=0x7f464c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab6e7b1616ee9365 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f465d600640 1 -- 172.21.15.132:0/2332005876 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c035020 con 0x7f466015b120 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f465d600640 1 -- 172.21.15.132:0/2332005876 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f464c031810 con 0x7f466015b120 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f465d600640 1 -- 172.21.15.132:0/2332005876 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c031be0 con 0x7f466015b120 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 -- 172.21.15.132:0/2332005876 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 msgr2=0x7f46600ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 --2- 172.21.15.132:0/2332005876 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f464c0097a0 tx=0x7f464c02b8b0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 -- 172.21.15.132:0/2332005876 shutdown_connections 2024-03-27T06:14:25.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 --2- 172.21.15.132:0/2332005876 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f466015b120 0x7f46600ce290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 --2- 172.21.15.132:0/2332005876 >> v2:172.21.15.132:3300/0 conn(0x7f4660158290 0x7f466015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 -- 172.21.15.132:0/2332005876 >> 172.21.15.132:0/2332005876 conn(0x7f4660067280 msgr2=0x7f4660067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 -- 172.21.15.132:0/2332005876 shutdown_connections 2024-03-27T06:14:25.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.592+0000 7f4665a00640 1 -- 172.21.15.132:0/2332005876 wait complete. 2024-03-27T06:14:25.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.595+0000 7f4665a00640 1 Processor -- start 2024-03-27T06:14:25.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.595+0000 7f4665a00640 1 -- start start 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 0x7f46600cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46601596e0 0x7f46601f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f466006e5d0 con 0x7f4660158290 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f466006f690 con 0x7f46600c9c80 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f4665a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f466006f1d0 con 0x7f46601596e0 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465e000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 0x7f46600cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51488/0 (socket says 172.21.15.132:51488) 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 -- 172.21.15.132:0/2209491679 learned_addr learned my addr 172.21.15.132:0/2209491679 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465f400640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46601596e0 0x7f46601f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 -- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46601596e0 msgr2=0x7f46601f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46601596e0 0x7f46601f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 -- 172.21.15.132:0/2209491679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 msgr2=0x7f46600cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 0x7f46600cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465ea00640 1 -- 172.21.15.132:0/2209491679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46600cc940 con 0x7f46600c9c80 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.597+0000 7f465e000640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 0x7f46600cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f465ea00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9e0 tx=0x7f465000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650017070 con 0x7f46600c9c80 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f465000c9d0 con 0x7f46600c9c80 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465000cdc0 con 0x7f46600c9c80 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46600ccc20 con 0x7f46600c9c80 2024-03-27T06:14:25.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.598+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46600cd1e0 con 0x7f46600c9c80 2024-03-27T06:14:25.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.599+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46600ca140 con 0x7f46600c9c80 2024-03-27T06:14:25.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.600+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f465001a070 con 0x7f46600c9c80 2024-03-27T06:14:25.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.601+0000 7f4656a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 0x7f464006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:25.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.601+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f465008b6e0 con 0x7f46600c9c80 2024-03-27T06:14:25.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.601+0000 7f465e000640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 0x7f464006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:25.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.602+0000 7f465e000640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 0x7f464006e9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f464c036000 tx=0x7f464c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:25.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.605+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4650059610 con 0x7f46600c9c80 2024-03-27T06:14:25.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.846+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4660068840 con 0x7f46600c9c80 2024-03-27T06:14:25.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.846+0000 7f4656a00640 1 -- 172.21.15.132:0/2209491679 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) ==== 53+0+73 (secure 0 0 0) 0x7f465005d910 con 0x7f46600c9c80 2024-03-27T06:14:25.837 INFO:teuthology.orchestra.run.smithi132.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-27T06:14:25.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 msgr2=0x7f464006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 0x7f464006e9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f464c036000 tx=0x7f464c03c040 comp rx=0 tx=0).stop 2024-03-27T06:14:25.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 msgr2=0x7f46601f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:25.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9e0 tx=0x7f465000eea0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.840 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 shutdown_connections 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46601596e0 0x7f46601f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4660158290 0x7f46600cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f464006c530 0x7f464006e9f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 --2- 172.21.15.132:0/2209491679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46600c9c80 0x7f46601f0850 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 >> 172.21.15.132:0/2209491679 conn(0x7f4660067280 msgr2=0x7f46600c89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 shutdown_connections 2024-03-27T06:14:25.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:25.849+0000 7f4665a00640 1 -- 172.21.15.132:0/2209491679 wait complete. 2024-03-27T06:14:25.860 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-27T06:14:25.860 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi132.front.sepia.ceph.com 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-27T06:14:25.861 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-27T06:14:25.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:25.861 DEBUG:teuthology.orchestra.run.smithi132:> ip addr 2024-03-27T06:14:25.907 INFO:teuthology.orchestra.run.smithi132.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: inet 127.0.0.1/8 scope host lo 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: valid_lft forever preferred_lft forever 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: inet6 ::1/128 scope host 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: valid_lft forever preferred_lft forever 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: link/ether 0c:c4:7a:d9:35:cc brd ff:ff:ff:ff:ff:ff 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: altname enp7s0f0 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: link/ether 0c:c4:7a:d9:35:cd brd ff:ff:ff:ff:ff:ff 2024-03-27T06:14:25.908 INFO:teuthology.orchestra.run.smithi132.stdout: altname enp7s0f1 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: link/ether 0c:c4:7a:88:77:3e brd ff:ff:ff:ff:ff:ff 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: link/ether 0c:c4:7a:88:77:3f brd ff:ff:ff:ff:ff:ff 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: inet 172.21.15.132/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: valid_lft 42508sec preferred_lft 42508sec 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: inet6 fe80::ec4:7aff:fe88:773f/64 scope link 2024-03-27T06:14:25.909 INFO:teuthology.orchestra.run.smithi132.stdout: valid_lft forever preferred_lft forever 2024-03-27T06:14:25.910 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-27T06:14:25.910 DEBUG:teuthology.orchestra.run.smithi132:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> set -e 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link add name ceph-brx type bridge 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip addr flush dev ceph-brx 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link set ceph-brx up 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-27T06:14:25.911 DEBUG:teuthology.orchestra.run.smithi132:> ') 2024-03-27T06:14:26.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:26.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:26.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:26.433 DEBUG:teuthology.orchestra.run.smithi132:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-27T06:14:26.508 INFO:teuthology.orchestra.run.smithi132.stdout:1 2024-03-27T06:14:26.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:26.512 DEBUG:teuthology.orchestra.run.smithi132:> route 2024-03-27T06:14:26.560 INFO:teuthology.orchestra.run.smithi132.stdout:Kernel IP routing table 2024-03-27T06:14:26.560 INFO:teuthology.orchestra.run.smithi132.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-27T06:14:26.560 INFO:teuthology.orchestra.run.smithi132.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-27T06:14:26.560 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-27T06:14:26.560 INFO:teuthology.orchestra.run.smithi132.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> set -e 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-27T06:14:26.561 DEBUG:teuthology.orchestra.run.smithi132:> ') 2024-03-27T06:14:26.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:27.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:27.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:27.294 DEBUG:teuthology.orchestra.run.smithi132:> ip netns list 2024-03-27T06:14:27.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:27.343 DEBUG:teuthology.orchestra.run.smithi132:> ip netns list-id 2024-03-27T06:14:27.393 DEBUG:teuthology.orchestra.run.smithi132:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-27T06:14:27.393 DEBUG:teuthology.orchestra.run.smithi132:> set -e 2024-03-27T06:14:27.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-27T06:14:27.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-27T06:14:27.393 DEBUG:teuthology.orchestra.run.smithi132:> ') 2024-03-27T06:14:27.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:27.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:27.583 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> set -e 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-27T06:14:27.583 DEBUG:teuthology.orchestra.run.smithi132:> ') 2024-03-27T06:14:27.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:27.908 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:27.917 DEBUG:teuthology.orchestra.run.smithi132:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-27T06:14:27.917 DEBUG:teuthology.orchestra.run.smithi132:> set -e 2024-03-27T06:14:27.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link set brx.0 up 2024-03-27T06:14:27.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link set dev brx.0 master ceph-brx 2024-03-27T06:14:27.918 DEBUG:teuthology.orchestra.run.smithi132:> ') 2024-03-27T06:14:28.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:28.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:28.118 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-27T06:14:28.118 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:28.597 INFO:teuthology.orchestra.run.smithi132.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-27T06:14:28.597 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-27T06:14:28.598 DEBUG:teuthology.orchestra.run.smithi132:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:28.647 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-27T06:14:28.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:28.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-27T06:14:28.764 INFO:teuthology.orchestra.run.smithi132.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-27T06:14:28.764 INFO:teuthology.orchestra.run.smithi132.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-27T06:14:29.457 INFO:teuthology.orchestra.run.smithi132.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-27T06:14:29.457 INFO:teuthology.orchestra.run.smithi132.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-27T06:14:29.457 INFO:teuthology.orchestra.run.smithi132.stdout:mount.ceph: resolved to: "172.21.15.132:3300,172.21.15.134:3300,172.21.15.134:3301" 2024-03-27T06:14:29.457 INFO:teuthology.orchestra.run.smithi132.stdout:mount.ceph: trying mount with new device syntax: 0@3dfa4fc8-5726-4b88-a1bf-7da524e3644e.cephfs=/ 2024-03-27T06:14:29.458 INFO:teuthology.orchestra.run.smithi132.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.132:3300/172.21.15.134:3300/172.21.15.134:3301" will pass to kernel 2024-03-27T06:14:29.465 INFO:teuthology.orchestra.run.smithi132.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-27T06:14:29.465 INFO:teuthology.orchestra.run.smithi132.stdout: You just mounted a file system that supports labels which does not 2024-03-27T06:14:29.465 INFO:teuthology.orchestra.run.smithi132.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-27T06:14:29.465 INFO:teuthology.orchestra.run.smithi132.stdout: applications will generate AVC messages and not be allowed access to 2024-03-27T06:14:29.466 INFO:teuthology.orchestra.run.smithi132.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-27T06:14:29.470 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-27T06:14:29.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:14:29.470 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:29.566 DEBUG:teuthology.orchestra.run.smithi132:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-27T06:14:29.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-27T06:14:29.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:29.694 INFO:teuthology.orchestra.run.smithi132.stdout:client4666 2024-03-27T06:14:29.695 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:14:29.695 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-27T06:14:29.745 DEBUG:teuthology.orchestra.run.smithi132:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-27T06:14:29.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-27T06:14:29.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:29.916 INFO:teuthology.orchestra.run.smithi132.stdout:client4666 2024-03-27T06:14:29.916 DEBUG:teuthology.orchestra.run.smithi132:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-27T06:14:30.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-27T06:14:30.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:30.058 INFO:teuthology.orchestra.run.smithi132.stdout:client4666 2024-03-27T06:14:30.058 DEBUG:teuthology.orchestra.run.smithi132:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3dfa4fc8-5726-4b88-a1bf-7da524e3644e.client4666/status') 2024-03-27T06:14:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-27T06:14:30.200 INFO:teuthology.orchestra.run.smithi132.stdout:instance: client.4666 (3)172.21.15.132:0/3657498388 2024-03-27T06:14:30.320 INFO:teuthology.orchestra.run.smithi132.stdout:blocklisted: false 2024-03-27T06:14:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:0+1 records in 2024-03-27T06:14:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:0+1 records out 2024-03-27T06:14:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:71 bytes copied, 7.7201e-05 s, 920 kB/s 2024-03-27T06:14:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-27T06:14:30.322 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-03-27T06:14:30.391 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:30.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.609+0000 7fae6fc00640 1 Processor -- start 2024-03-27T06:14:30.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.609+0000 7fae6fc00640 1 -- start start 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6fc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fae68152950 0x7fae681590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6fc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fae68073550 0x7fae68071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6fc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae6806b630 con 0x7fae68073970 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6fc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fae68152d30 con 0x7fae68073550 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6fc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fae6806b8d0 con 0x7fae68152950 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6d200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fae68152950 0x7fae681590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae67e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fae68073550 0x7fae68071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6d200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fae68152950 0x7fae681590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51492/0 (socket says 172.21.15.132:51492) 2024-03-27T06:14:30.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.611+0000 7fae6d200640 1 -- 172.21.15.132:0/3101151227 learned_addr learned my addr 172.21.15.132:0/3101151227 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:30.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.612+0000 7fae6d200640 1 -- 172.21.15.132:0/3101151227 >> v2:172.21.15.132:3300/0 conn(0x7fae68073550 msgr2=0x7fae68071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:30.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.612+0000 7fae6d200640 1 --2- 172.21.15.132:0/3101151227 >> v2:172.21.15.132:3300/0 conn(0x7fae68073550 0x7fae68071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.612+0000 7fae6d200640 1 -- 172.21.15.132:0/3101151227 >> v1:172.21.15.132:6789/0 conn(0x7fae68073970 legacy=0x7fae6806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.612+0000 7fae6d200640 1 -- 172.21.15.132:0/3101151227 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6806bbb0 con 0x7fae68152950 2024-03-27T06:14:30.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.613+0000 7fae6d200640 1 --2- 172.21.15.132:0/3101151227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68152950 0x7fae681590a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009870 tx=0x7fae5c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75959d298cad6b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:30.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.613+0000 7fae67400640 1 -- 172.21.15.132:0/3101151227 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c034070 con 0x7fae68152950 2024-03-27T06:14:30.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.613+0000 7fae67400640 1 -- 172.21.15.132:0/3101151227 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c030860 con 0x7fae68152950 2024-03-27T06:14:30.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.613+0000 7fae67400640 1 -- 172.21.15.132:0/3101151227 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c030c30 con 0x7fae68152950 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.613+0000 7fae6fc00640 1 -- 172.21.15.132:0/3101151227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68152950 msgr2=0x7fae681590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 --2- 172.21.15.132:0/3101151227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68152950 0x7fae681590a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009870 tx=0x7fae5c02b9a0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 -- 172.21.15.132:0/3101151227 shutdown_connections 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 --2- 172.21.15.132:0/3101151227 >> v2:172.21.15.132:3300/0 conn(0x7fae68073550 0x7fae68071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 --2- 172.21.15.132:0/3101151227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68152950 0x7fae681590a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 -- 172.21.15.132:0/3101151227 >> 172.21.15.132:0/3101151227 conn(0x7fae680cc060 msgr2=0x7fae680cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 -- 172.21.15.132:0/3101151227 shutdown_connections 2024-03-27T06:14:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.614+0000 7fae6fc00640 1 -- 172.21.15.132:0/3101151227 wait complete. 2024-03-27T06:14:30.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.615+0000 7fae6fc00640 1 Processor -- start 2024-03-27T06:14:30.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.616+0000 7fae6fc00640 1 -- start start 2024-03-27T06:14:30.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 0x7fae6815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae68152950 0x7fae680aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fae6806c450 con 0x7fae68152950 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fae6806c170 con 0x7fae68073970 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6fc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fae6806d660 con 0x7fae68073550 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae67e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 0x7fae6815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57552/0 (socket says 172.21.15.132:57552) 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 -- 172.21.15.132:0/1639862273 learned_addr learned my addr 172.21.15.132:0/1639862273 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6dc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae68152950 0x7fae680aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 -- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 msgr2=0x7fae6815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 0x7fae6815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 -- 172.21.15.132:0/1639862273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae68152950 msgr2=0x7fae680aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae68152950 0x7fae680aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae6d200640 1 -- 172.21.15.132:0/1639862273 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae681e1bd0 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.617+0000 7fae67e00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 0x7fae6815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae6d200640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae5c02b9e0 tx=0x7fae5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c034040 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c039d90 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c043630 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae681f0c30 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.618+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae680c37e0 con 0x7fae68073550 2024-03-27T06:14:30.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.620+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae68073170 con 0x7fae68073550 2024-03-27T06:14:30.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.620+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fae5c04a060 con 0x7fae68073550 2024-03-27T06:14:30.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.621+0000 7fae64c00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 0x7fae4806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:30.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.621+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae5c0ad5c0 con 0x7fae68073550 2024-03-27T06:14:30.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.621+0000 7fae67e00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 0x7fae4806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:30.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.622+0000 7fae67e00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 0x7fae4806e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae68074780 tx=0x7fae54009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:30.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.626+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fae68073170 con 0x7fae68073550 2024-03-27T06:14:30.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.939+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae680c6750 con 0x7fae68073550 2024-03-27T06:14:30.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.940+0000 7fae64c00640 1 -- 172.21.15.132:0/1639862273 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fae680c6750 con 0x7fae68073550 2024-03-27T06:14:30.931 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:30.931 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.942+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 msgr2=0x7fae4806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.942+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 0x7fae4806e9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae68074780 tx=0x7fae54009190 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 msgr2=0x7fae680c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae5c02b9e0 tx=0x7fae5c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 shutdown_connections 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fae68152950 0x7fae680aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fae4806c530 0x7fae4806e9f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae68073970 0x7fae6815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 --2- 172.21.15.132:0/1639862273 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fae68073550 0x7fae680c45a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 >> 172.21.15.132:0/1639862273 conn(0x7fae680cc060 msgr2=0x7fae680c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 shutdown_connections 2024-03-27T06:14:30.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:30.943+0000 7fae6fc00640 1 -- 172.21.15.132:0/1639862273 wait complete. 2024-03-27T06:14:30.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:14:30.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:14:31.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.164+0000 7f3904200640 1 Processor -- start 2024-03-27T06:14:31.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.164+0000 7f3904200640 1 -- start start 2024-03-27T06:14:31.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.165+0000 7f3904200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f38fc154f80 0x7f38fc15b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3904200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc15bb40 con 0x7f38fc0cb2f0 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3904200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38fc0ccd30 con 0x7f38fc0c8bd0 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3904200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f38fc155360 con 0x7f38fc154f80 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3901800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f38fc154f80 0x7f38fc15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3902200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f38fc0c8bd0 0x7f38fc0c9020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:59836/0 (socket says 172.21.15.132:59836) 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3902200640 1 -- 172.21.15.132:0/3397572514 learned_addr learned my addr 172.21.15.132:0/3397572514 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3901800640 1 -- 172.21.15.132:0/3397572514 >> v1:172.21.15.134:6789/0 conn(0x7f38fc0c8bd0 legacy=0x7f38fc0c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3901800640 1 -- 172.21.15.132:0/3397572514 >> v1:172.21.15.132:6789/0 conn(0x7f38fc0cb2f0 legacy=0x7f38fc0d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.166+0000 7f3901800640 1 -- 172.21.15.132:0/3397572514 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc0ccfd0 con 0x7f38fc154f80 2024-03-27T06:14:31.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.167+0000 7f3901800640 1 --2- 172.21.15.132:0/3397572514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc15b600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009720 tx=0x7f38f002b6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3149ee5f945a7bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f38fbe00640 1 -- 172.21.15.132:0/3397572514 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0037070 con 0x7f38fc154f80 2024-03-27T06:14:31.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f38fbe00640 1 -- 172.21.15.132:0/3397572514 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f0030710 con 0x7f38fc154f80 2024-03-27T06:14:31.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f38fbe00640 1 -- 172.21.15.132:0/3397572514 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0030ae0 con 0x7f38fc154f80 2024-03-27T06:14:31.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 -- 172.21.15.132:0/3397572514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 msgr2=0x7f38fc15b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 --2- 172.21.15.132:0/3397572514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc15b600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009720 tx=0x7f38f002b6a0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 -- 172.21.15.132:0/3397572514 shutdown_connections 2024-03-27T06:14:31.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 --2- 172.21.15.132:0/3397572514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc15b600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 -- 172.21.15.132:0/3397572514 >> 172.21.15.132:0/3397572514 conn(0x7f38fc06d9f0 msgr2=0x7f38fc06de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:31.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 -- 172.21.15.132:0/3397572514 shutdown_connections 2024-03-27T06:14:31.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.168+0000 7f3904200640 1 -- 172.21.15.132:0/3397572514 wait complete. 2024-03-27T06:14:31.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.170+0000 7f3904200640 1 Processor -- start 2024-03-27T06:14:31.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.170+0000 7f3904200640 1 -- start start 2024-03-27T06:14:31.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 0x7f38fc15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38fc0cb2f0 0x7f38fc14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc0cd870 con 0x7f38fc0cb2f0 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc0cd590 con 0x7f38fc154f80 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3904200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f38fc0cc0a0 con 0x7f38fc0c8bd0 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3902200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3901800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 0x7f38fc15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3902200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51512/0 (socket says 172.21.15.132:51512) 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3902200640 1 -- 172.21.15.132:0/429045926 learned_addr learned my addr 172.21.15.132:0/429045926 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3901800640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 0x7f38fc15eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57566/0 (socket says 172.21.15.132:57566) 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.171+0000 7f3900e00640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38fc0cb2f0 0x7f38fc14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 -- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 msgr2=0x7f38fc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 0x7f38fc15eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 -- 172.21.15.132:0/429045926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38fc0cb2f0 msgr2=0x7f38fc14f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38fc0cb2f0 0x7f38fc14f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 -- 172.21.15.132:0/429045926 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc14f2c0 con 0x7f38fc154f80 2024-03-27T06:14:31.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3902200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f38f400cb50 tx=0x7f38f400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4018070 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f400ce50 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc15e7d0 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f400ab40 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.172+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38fc151c90 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.173+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38fc068950 con 0x7f38fc154f80 2024-03-27T06:14:31.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.174+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f38f401b070 con 0x7f38fc154f80 2024-03-27T06:14:31.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.175+0000 7f38f9600640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 0x7f38dc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.175+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f38f408add0 con 0x7f38fc154f80 2024-03-27T06:14:31.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.175+0000 7f3901800640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 0x7f38dc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.176+0000 7f3901800640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 0x7f38dc06e9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38f0034000 tx=0x7f38f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.180+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f38f4058da0 con 0x7f38fc154f80 2024-03-27T06:14:31.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.377+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38fc0cbe80 con 0x7f38fc154f80 2024-03-27T06:14:31.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.377+0000 7f38f9600640 1 -- 172.21.15.132:0/429045926 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f38f408c020 con 0x7f38fc154f80 2024-03-27T06:14:31.368 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 msgr2=0x7f38dc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 0x7f38dc06e9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38f0034000 tx=0x7f38f0008030 comp rx=0 tx=0).stop 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 msgr2=0x7f38fc14fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f38f400cb50 tx=0x7f38f400a430 comp rx=0 tx=0).stop 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 shutdown_connections 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f38dc06c530 0x7f38dc06e9f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38fc154f80 0x7f38fc14fcb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38fc0cb2f0 0x7f38fc14f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 --2- 172.21.15.132:0/429045926 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f38fc0c8bd0 0x7f38fc15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.372 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 >> 172.21.15.132:0/429045926 conn(0x7f38fc06d9f0 msgr2=0x7f38fc0ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:31.372 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 shutdown_connections 2024-03-27T06:14:31.372 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.380+0000 7f3904200640 1 -- 172.21.15.132:0/429045926 wait complete. 2024-03-27T06:14:31.389 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:14:31.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.588+0000 7f185f200640 1 Processor -- start 2024-03-27T06:14:31.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.588+0000 7f185f200640 1 -- start start 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185f200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1858072d90 0x7f18580c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185f200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f185806c360 con 0x7f18580733c0 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185f200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1858073840 con 0x7f1858154c90 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185f200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f1858073b20 con 0x7f1858072d90 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f1857e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1858154c90 0x7f185815b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:59844/0 (socket says 172.21.15.132:59844) 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f1857e00640 1 -- 172.21.15.132:0/2841788741 learned_addr learned my addr 172.21.15.132:0/2841788741 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185d200640 1 --2- 172.21.15.132:0/2841788741 >> v2:172.21.15.134:3300/0 conn(0x7f1858072d90 0x7f18580c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185d200640 1 -- 172.21.15.132:0/2841788741 >> v1:172.21.15.134:6789/0 conn(0x7f1858154c90 legacy=0x7f185815b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f1856a00640 1 -- 172.21.15.132:0/2841788741 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262373983 0 0) 0x7f1858073840 con 0x7f1858154c90 2024-03-27T06:14:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185d200640 1 -- 172.21.15.132:0/2841788741 >> v1:172.21.15.132:6789/0 conn(0x7f18580733c0 legacy=0x7f18580cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.591+0000 7f185d200640 1 -- 172.21.15.132:0/2841788741 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185806bda0 con 0x7f1858072d90 2024-03-27T06:14:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.592+0000 7f185d200640 1 --2- 172.21.15.132:0/2841788741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1858072d90 0x7f18580c9fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1850009d90 tx=0x7f185002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=876885b8a0a66745 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f1856a00640 1 -- 172.21.15.132:0/2841788741 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850039070 con 0x7f1858072d90 2024-03-27T06:14:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f1856a00640 1 -- 172.21.15.132:0/2841788741 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f185002cc60 con 0x7f1858072d90 2024-03-27T06:14:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f1856a00640 1 -- 172.21.15.132:0/2841788741 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850034660 con 0x7f1858072d90 2024-03-27T06:14:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 -- 172.21.15.132:0/2841788741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1858072d90 msgr2=0x7f18580c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 --2- 172.21.15.132:0/2841788741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1858072d90 0x7f18580c9fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1850009d90 tx=0x7f185002beb0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 -- 172.21.15.132:0/2841788741 shutdown_connections 2024-03-27T06:14:31.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 --2- 172.21.15.132:0/2841788741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1858072d90 0x7f18580c9fe0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 -- 172.21.15.132:0/2841788741 >> 172.21.15.132:0/2841788741 conn(0x7f185806c630 msgr2=0x7f185806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:31.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 -- 172.21.15.132:0/2841788741 shutdown_connections 2024-03-27T06:14:31.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.593+0000 7f185f200640 1 -- 172.21.15.132:0/2841788741 wait complete. 2024-03-27T06:14:31.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.595+0000 7f185f200640 1 Processor -- start 2024-03-27T06:14:31.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.595+0000 7f185f200640 1 -- start start 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18580733c0 0x7f18581f1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1858154c90 0x7f185814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1858155070 con 0x7f1858154c90 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f18580c9b50 con 0x7f18580733c0 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185f200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f18580cd710 con 0x7f1858072d90 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f1857e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f185d200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1858154c90 0x7f185814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f1857e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:57574/0 (socket says 172.21.15.132:57574) 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f1857400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18580733c0 0x7f18581f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.597+0000 7f1857e00640 1 -- 172.21.15.132:0/351838736 learned_addr learned my addr 172.21.15.132:0/351838736 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:31.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 -- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18580733c0 msgr2=0x7f18581f1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18580733c0 0x7f18581f1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 -- 172.21.15.132:0/351838736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1858154c90 msgr2=0x7f185814db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1858154c90 0x7f185814db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 -- 172.21.15.132:0/351838736 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18581f1f90 con 0x7f1858072d90 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f1857e00640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f184800c4a0 tx=0x7f184800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848018070 con 0x7f1858072d90 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18480074e0 con 0x7f1858072d90 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848004a20 con 0x7f1858072d90 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18581f1570 con 0x7f1858072d90 2024-03-27T06:14:31.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.598+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1858151000 con 0x7f1858072d90 2024-03-27T06:14:31.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.599+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1858068950 con 0x7f1858072d90 2024-03-27T06:14:31.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.601+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f184801b070 con 0x7f1858072d90 2024-03-27T06:14:31.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.602+0000 7f184fe00640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 0x7f183806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:31.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.602+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f184808a910 con 0x7f1858072d90 2024-03-27T06:14:31.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.602+0000 7f1857400640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 0x7f183806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:31.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.603+0000 7f1857400640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 0x7f183806e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18580c9090 tx=0x7f1844009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:31.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.607+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f18480588e0 con 0x7f1858072d90 2024-03-27T06:14:31.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.882+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1858072830 con 0x7f1858072d90 2024-03-27T06:14:31.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.883+0000 7f184fe00640 1 -- 172.21.15.132:0/351838736 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f184805cbe0 con 0x7f1858072d90 2024-03-27T06:14:31.874 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:14:31.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 msgr2=0x7f183806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 0x7f183806e9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18580c9090 tx=0x7f1844009040 comp rx=0 tx=0).stop 2024-03-27T06:14:31.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 msgr2=0x7f18581f1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:31.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f184800c4a0 tx=0x7f184800c960 comp rx=0 tx=0).stop 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 shutdown_connections 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1858154c90 0x7f185814db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f183806c530 0x7f183806e9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18580733c0 0x7f18581f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 --2- 172.21.15.132:0/351838736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1858072d90 0x7f18581f1a50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 >> 172.21.15.132:0/351838736 conn(0x7f185806c630 msgr2=0x7f18581530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 shutdown_connections 2024-03-27T06:14:31.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:31.885+0000 7f185f200640 1 -- 172.21.15.132:0/351838736 wait complete. 2024-03-27T06:14:31.893 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:14:32.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.125+0000 7fe199c00640 1 Processor -- start 2024-03-27T06:14:32.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.126+0000 7fe199c00640 1 -- start start 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe199c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe1941517c0 0x7fe1940cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe199c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe1940c5110 0x7fe194165090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe199c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1940cafe0 con 0x7fe1940cc2b0 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe199c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fe1940cc770 con 0x7fe1940c5110 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe199c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fe1940ccd30 con 0x7fe1941517c0 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe193e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe1941517c0 0x7fe1940cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe193400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe1940cc2b0 0x7fe1940d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:55758/0 (socket says 172.21.15.132:55758) 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.127+0000 7fe193400640 1 -- 172.21.15.132:0/3367898403 learned_addr learned my addr 172.21.15.132:0/3367898403 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe199200640 1 --2- 172.21.15.132:0/3367898403 >> v2:172.21.15.132:3300/0 conn(0x7fe1940c5110 0x7fe194165090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe193e00640 1 -- 172.21.15.132:0/3367898403 >> v2:172.21.15.132:3300/0 conn(0x7fe1940c5110 msgr2=0x7fe194165090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe193e00640 1 --2- 172.21.15.132:0/3367898403 >> v2:172.21.15.132:3300/0 conn(0x7fe1940c5110 0x7fe194165090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe193e00640 1 -- 172.21.15.132:0/3367898403 >> v1:172.21.15.132:6789/0 conn(0x7fe1940cc2b0 legacy=0x7fe1940d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe193e00640 1 -- 172.21.15.132:0/3367898403 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1940cb2c0 con 0x7fe1941517c0 2024-03-27T06:14:32.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.128+0000 7fe192a00640 1 -- 172.21.15.132:0/3367898403 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416868490 0 0) 0x7fe1940cafe0 con 0x7fe1940cc2b0 2024-03-27T06:14:32.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.130+0000 7fe193e00640 1 --2- 172.21.15.132:0/3367898403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1941517c0 0x7fe1940cd1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65db22e09432a71e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.130+0000 7fe192a00640 1 -- 172.21.15.132:0/3367898403 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184039070 con 0x7fe1941517c0 2024-03-27T06:14:32.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.130+0000 7fe192a00640 1 -- 172.21.15.132:0/3367898403 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18402cc60 con 0x7fe1941517c0 2024-03-27T06:14:32.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.130+0000 7fe192a00640 1 -- 172.21.15.132:0/3367898403 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184034660 con 0x7fe1941517c0 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 -- 172.21.15.132:0/3367898403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1941517c0 msgr2=0x7fe1940cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 --2- 172.21.15.132:0/3367898403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1941517c0 0x7fe1940cd1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe184009d90 tx=0x7fe18402beb0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 -- 172.21.15.132:0/3367898403 shutdown_connections 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 --2- 172.21.15.132:0/3367898403 >> v2:172.21.15.132:3300/0 conn(0x7fe1940c5110 0x7fe194165090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 --2- 172.21.15.132:0/3367898403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1941517c0 0x7fe1940cd1c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 -- 172.21.15.132:0/3367898403 >> 172.21.15.132:0/3367898403 conn(0x7fe19406dc80 msgr2=0x7fe19406e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 -- 172.21.15.132:0/3367898403 shutdown_connections 2024-03-27T06:14:32.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.131+0000 7fe199c00640 1 -- 172.21.15.132:0/3367898403 wait complete. 2024-03-27T06:14:32.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.133+0000 7fe199c00640 1 Processor -- start 2024-03-27T06:14:32.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.133+0000 7fe199c00640 1 -- start start 2024-03-27T06:14:32.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 0x7fe1940b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 0x7fe19414cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940d0c10 con 0x7fe1940c5110 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940c5660 con 0x7fe1940cc2b0 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe1940d0d90 con 0x7fe1941517c0 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 0x7fe1940b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52762/0 (socket says 172.21.15.132:52762) 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 -- 172.21.15.132:0/2557066675 learned_addr learned my addr 172.21.15.132:0/2557066675 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 0x7fe19414cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 -- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 msgr2=0x7fe19414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 0x7fe19414cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 -- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 msgr2=0x7fe1940b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 0x7fe1940b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193e00640 1 -- 172.21.15.132:0/2557066675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1940b43f0 con 0x7fe1940c5110 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe193400640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 0x7fe1940b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.134+0000 7fe199200640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 0x7fe19414cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe193e00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe184037040 tx=0x7fe184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184039050 con 0x7fe1940c5110 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe184045070 con 0x7fe1940c5110 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18403e850 con 0x7fe1940c5110 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1940b46d0 con 0x7fe1940c5110 2024-03-27T06:14:32.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.135+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1940b4c90 con 0x7fe1940c5110 2024-03-27T06:14:32.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.136+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1940686c0 con 0x7fe1940c5110 2024-03-27T06:14:32.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.137+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe18403e9f0 con 0x7fe1940c5110 2024-03-27T06:14:32.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.139+0000 7fe18be00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 0x7fe17006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.139+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe1840ae620 con 0x7fe1940c5110 2024-03-27T06:14:32.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.139+0000 7fe193400640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 0x7fe17006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.139+0000 7fe193400640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 0x7fe17006e9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1941ee300 tx=0x7fe17c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.143+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe18407c710 con 0x7fe1940c5110 2024-03-27T06:14:32.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.338+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe194154e60 con 0x7fe1940c5110 2024-03-27T06:14:32.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.339+0000 7fe18be00640 1 -- 172.21.15.132:0/2557066675 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fe184080a10 con 0x7fe1940c5110 2024-03-27T06:14:32.329 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:14:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 msgr2=0x7fe17006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 0x7fe17006e9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1941ee300 tx=0x7fe17c007d50 comp rx=0 tx=0).stop 2024-03-27T06:14:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 msgr2=0x7fe19414ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe184037040 tx=0x7fe184004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:32.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 shutdown_connections 2024-03-27T06:14:32.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe1941517c0 0x7fe19414cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe17006c530 0x7fe17006e9f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1940cc2b0 0x7fe1940b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 --2- 172.21.15.132:0/2557066675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1940c5110 0x7fe19414ada0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 >> 172.21.15.132:0/2557066675 conn(0x7fe19406dc80 msgr2=0x7fe19414ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:32.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 shutdown_connections 2024-03-27T06:14:32.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.341+0000 7fe199c00640 1 -- 172.21.15.132:0/2557066675 wait complete. 2024-03-27T06:14:32.351 INFO:tasks.mds_thrash:mds thrasher using random seed: 1711520072 2024-03-27T06:14:32.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:32.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.564+0000 7f712f200640 1 Processor -- start 2024-03-27T06:14:32.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.565+0000 7f712f200640 1 -- start start 2024-03-27T06:14:32.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f712f200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f712f200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7128151ad0 0x7f71280c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f712f200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f712806b520 con 0x7f7128073880 2024-03-27T06:14:32.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f712f200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f7128072be0 con 0x7f7128151ad0 2024-03-27T06:14:32.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f712f200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f712806b7c0 con 0x7f71281516f0 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f7127e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f711fe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7128151ad0 0x7f71280c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f7127e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34682/0 (socket says 172.21.15.132:34682) 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.566+0000 7f7127e00640 1 -- 172.21.15.132:0/1583715754 learned_addr learned my addr 172.21.15.132:0/1583715754 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.567+0000 7f7127e00640 1 -- 172.21.15.132:0/1583715754 >> v2:172.21.15.132:3300/0 conn(0x7f7128151ad0 msgr2=0x7f71280c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.567+0000 7f7127e00640 1 --2- 172.21.15.132:0/1583715754 >> v2:172.21.15.132:3300/0 conn(0x7f7128151ad0 0x7f71280c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.567+0000 7f7127e00640 1 -- 172.21.15.132:0/1583715754 >> v1:172.21.15.132:6789/0 conn(0x7f7128073880 legacy=0x7f7128157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.567+0000 7f7127e00640 1 -- 172.21.15.132:0/1583715754 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712806baa0 con 0x7f71281516f0 2024-03-27T06:14:32.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f7127e00640 1 --2- 172.21.15.132:0/1583715754 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7118009870 tx=0x7f711802dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a85ffda3cab2eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f7127400640 1 -- 172.21.15.132:0/1583715754 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711803b070 con 0x7f71281516f0 2024-03-27T06:14:32.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f7127400640 1 -- 172.21.15.132:0/1583715754 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711802e9e0 con 0x7f71281516f0 2024-03-27T06:14:32.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f7127400640 1 -- 172.21.15.132:0/1583715754 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711802edb0 con 0x7f71281516f0 2024-03-27T06:14:32.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f712f200640 1 -- 172.21.15.132:0/1583715754 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 msgr2=0x7f71280c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.569+0000 7f712f200640 1 --2- 172.21.15.132:0/1583715754 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7118009870 tx=0x7f711802dc30 comp rx=0 tx=0).stop 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 -- 172.21.15.132:0/1583715754 shutdown_connections 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 --2- 172.21.15.132:0/1583715754 >> v2:172.21.15.132:3300/0 conn(0x7f7128151ad0 0x7f71280c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 --2- 172.21.15.132:0/1583715754 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f71280c7e20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 -- 172.21.15.132:0/1583715754 >> 172.21.15.132:0/1583715754 conn(0x7f71280cc060 msgr2=0x7f71280cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 -- 172.21.15.132:0/1583715754 shutdown_connections 2024-03-27T06:14:32.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.570+0000 7f712f200640 1 -- 172.21.15.132:0/1583715754 wait complete. 2024-03-27T06:14:32.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.571+0000 7f712f200640 1 Processor -- start 2024-03-27T06:14:32.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.571+0000 7f712f200640 1 -- start start 2024-03-27T06:14:32.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 0x7f7128156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7128151ad0 0x7f71281f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f712806bd80 con 0x7f7128073880 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7128072870 con 0x7f7128151ad0 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.572+0000 7f712f200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f71280c1080 con 0x7f71281516f0 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f7127e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 0x7f7128156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f7127e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 0x7f7128156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52778/0 (socket says 172.21.15.132:52778) 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34698/0 (socket says 172.21.15.132:34698) 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f7127e00640 1 -- 172.21.15.132:0/2121811496 learned_addr learned my addr 172.21.15.132:0/2121811496 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f712d200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7128151ad0 0x7f71281f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 -- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7128151ad0 msgr2=0x7f71281f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7128151ad0 0x7f71281f4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 -- 172.21.15.132:0/2121811496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 msgr2=0x7f7128156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 0x7f7128156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 -- 172.21.15.132:0/2121811496 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71281f4a90 con 0x7f71281516f0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.573+0000 7f711fe00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f711000e840 tx=0x7f711000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.574+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110017070 con 0x7f71281516f0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.574+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711000c730 con 0x7f71281516f0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.574+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71281451f0 con 0x7f71281516f0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.574+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711000cb00 con 0x7f71281516f0 2024-03-27T06:14:32.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.574+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71281457b0 con 0x7f71281516f0 2024-03-27T06:14:32.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.575+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7128068840 con 0x7f71281516f0 2024-03-27T06:14:32.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.576+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f711001a070 con 0x7f71281516f0 2024-03-27T06:14:32.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.577+0000 7f7124c00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 0x7f710406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:32.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.577+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f711008b4d0 con 0x7f71281516f0 2024-03-27T06:14:32.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.578+0000 7f7127e00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 0x7f710406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:32.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.578+0000 7f7127e00640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 0x7f710406e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f71180090b0 tx=0x7f7118032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:32.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.585+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f71281f4a90 con 0x7f71281516f0 2024-03-27T06:14:32.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.787+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71280c19a0 con 0x7f71281516f0 2024-03-27T06:14:32.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.788+0000 7f7124c00640 1 -- 172.21.15.132:0/2121811496 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f7110059400 con 0x7f71281516f0 2024-03-27T06:14:32.780 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:32.780 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:32.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 msgr2=0x7f710406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 0x7f710406e9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f71180090b0 tx=0x7f7118032000 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 msgr2=0x7f7128144050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f711000e840 tx=0x7f711000ed00 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 shutdown_connections 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f710406c530 0x7f710406e9f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7128151ad0 0x7f71281f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f71281516f0 0x7f7128144050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 --2- 172.21.15.132:0/2121811496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7128073880 0x7f7128156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 >> 172.21.15.132:0/2121811496 conn(0x7f71280cc060 msgr2=0x7f712814eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 shutdown_connections 2024-03-27T06:14:32.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:32.791+0000 7f712f200640 1 -- 172.21.15.132:0/2121811496 wait complete. 2024-03-27T06:14:32.802 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:14:32.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:14:32.904 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:14:32.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:14:33.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.022+0000 7ff4c1200640 1 Processor -- start 2024-03-27T06:14:33.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.022+0000 7ff4c1200640 1 -- start start 2024-03-27T06:14:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4c1200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4c1200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff4bc0caca0 0x7ff4bc0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4c1200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4bc15ab10 con 0x7ff4bc0cff60 2024-03-27T06:14:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4c1200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc15b5f0 con 0x7ff4bc0caca0 2024-03-27T06:14:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4c1200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc15adb0 con 0x7ff4bc15a640 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4baa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4bb400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ff4bc0cff60 0x7ff4bc0cc100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:55778/0 (socket says 172.21.15.132:55778) 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4bb400640 1 -- 172.21.15.132:0/3824470112 learned_addr learned my addr 172.21.15.132:0/3824470112 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.025+0000 7ff4bbe00640 1 --2- 172.21.15.132:0/3824470112 >> v2:172.21.15.132:3300/0 conn(0x7ff4bc0caca0 0x7ff4bc0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.026+0000 7ff4baa00640 1 -- 172.21.15.132:0/3824470112 >> v2:172.21.15.132:3300/0 conn(0x7ff4bc0caca0 msgr2=0x7ff4bc0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.026+0000 7ff4baa00640 1 --2- 172.21.15.132:0/3824470112 >> v2:172.21.15.132:3300/0 conn(0x7ff4bc0caca0 0x7ff4bc0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.026+0000 7ff4baa00640 1 -- 172.21.15.132:0/3824470112 >> v1:172.21.15.132:6789/0 conn(0x7ff4bc0cff60 legacy=0x7ff4bc0cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.026+0000 7ff4baa00640 1 -- 172.21.15.132:0/3824470112 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc15b050 con 0x7ff4bc15a640 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.027+0000 7ff4baa00640 1 --2- 172.21.15.132:0/3824470112 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0d0340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4087cb242471f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.027+0000 7ff4ba000640 1 -- 172.21.15.132:0/3824470112 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8034070 con 0x7ff4bc15a640 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.027+0000 7ff4ba000640 1 -- 172.21.15.132:0/3824470112 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a8030800 con 0x7ff4bc15a640 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.027+0000 7ff4ba000640 1 -- 172.21.15.132:0/3824470112 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8030bd0 con 0x7ff4bc15a640 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 -- 172.21.15.132:0/3824470112 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 msgr2=0x7ff4bc0d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3824470112 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0d0340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b940 comp rx=0 tx=0).stop 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 -- 172.21.15.132:0/3824470112 shutdown_connections 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3824470112 >> v2:172.21.15.132:3300/0 conn(0x7ff4bc0caca0 0x7ff4bc0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3824470112 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0d0340 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 -- 172.21.15.132:0/3824470112 >> 172.21.15.132:0/3824470112 conn(0x7ff4bc072e30 msgr2=0x7ff4bc067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 -- 172.21.15.132:0/3824470112 shutdown_connections 2024-03-27T06:14:33.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.028+0000 7ff4c1200640 1 -- 172.21.15.132:0/3824470112 wait complete. 2024-03-27T06:14:33.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.029+0000 7ff4c1200640 1 Processor -- start 2024-03-27T06:14:33.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.030+0000 7ff4c1200640 1 -- start start 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.030+0000 7ff4c1200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4bc0caca0 0x7ff4bc1f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.030+0000 7ff4c1200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4c1200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4c1200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc154c20 con 0x7ff4bc0cff60 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4c1200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc0ccaf0 con 0x7ff4bc0caca0 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4c1200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc1e5010 con 0x7ff4bc15a640 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4baa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34726/0 (socket says 172.21.15.132:34726) 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 -- 172.21.15.132:0/3019960628 learned_addr learned my addr 172.21.15.132:0/3019960628 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4baa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52790/0 (socket says 172.21.15.132:52790) 2024-03-27T06:14:33.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bb400640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4bc0caca0 0x7ff4bc1f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 -- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4bc0caca0 msgr2=0x7ff4bc1f1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4bc0caca0 0x7ff4bc1f1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 -- 172.21.15.132:0/3019960628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 msgr2=0x7ff4bc0b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4bbe00640 1 -- 172.21.15.132:0/3019960628 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc0b4240 con 0x7ff4bc15a640 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.031+0000 7ff4baa00640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4bbe00640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009dd0 tx=0x7ff4b400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4018020 con 0x7ff4bc15a640 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4014480 con 0x7ff4bc15a640 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4014870 con 0x7ff4bc15a640 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4bc1f1bd0 con 0x7ff4bc15a640 2024-03-27T06:14:33.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.032+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4bc1e6530 con 0x7ff4bc15a640 2024-03-27T06:14:33.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.033+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4bc0758a0 con 0x7ff4bc15a640 2024-03-27T06:14:33.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.034+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff4b4005d10 con 0x7ff4bc15a640 2024-03-27T06:14:33.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.035+0000 7ff4b3400640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 0x7ff49c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.035+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff4b408c0c0 con 0x7ff4bc15a640 2024-03-27T06:14:33.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.035+0000 7ff4bb400640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 0x7ff49c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.036+0000 7ff4bb400640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 0x7ff49c06e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000f60 tx=0x7ff4ac007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.039+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff4b4022070 con 0x7ff4bc15a640 2024-03-27T06:14:33.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.241+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4bc153c40 con 0x7ff4bc15a640 2024-03-27T06:14:33.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.242+0000 7ff4b3400640 1 -- 172.21.15.132:0/3019960628 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ff4b4010040 con 0x7ff4bc15a640 2024-03-27T06:14:33.233 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:14:33.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 msgr2=0x7ff49c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 0x7ff49c06e9f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000f60 tx=0x7ff4ac007e80 comp rx=0 tx=0).stop 2024-03-27T06:14:33.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 msgr2=0x7ff4bc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009dd0 tx=0x7ff4b400c490 comp rx=0 tx=0).stop 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 shutdown_connections 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff4bc15a640 0x7ff4bc0b4f30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4bc0cff60 0x7ff4bc0b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff49c06c530 0x7ff49c06e9f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 --2- 172.21.15.132:0/3019960628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4bc0caca0 0x7ff4bc1f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 >> 172.21.15.132:0/3019960628 conn(0x7ff4bc072e30 msgr2=0x7ff4bc067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 shutdown_connections 2024-03-27T06:14:33.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.244+0000 7ff4c1200640 1 -- 172.21.15.132:0/3019960628 wait complete. 2024-03-27T06:14:33.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:14:33.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.506+0000 7f98ef200640 1 Processor -- start 2024-03-27T06:14:33.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.506+0000 7f98ef200640 1 -- start start 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.507+0000 7f98ef200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.507+0000 7f98ef200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 0x7f98e80d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ef200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e80cafe0 con 0x7f98e80c5110 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ef200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f98e80cc770 con 0x7f98e80cc2b0 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ef200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f98e80ccd30 con 0x7f98e81517c0 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34740/0 (socket says 172.21.15.132:34740) 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 -- 172.21.15.132:0/3586589254 learned_addr learned my addr 172.21.15.132:0/3586589254 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ed400640 1 --2- 172.21.15.132:0/3586589254 >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 0x7f98e80d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 -- 172.21.15.132:0/3586589254 >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 msgr2=0x7f98e80d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:33.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 --2- 172.21.15.132:0/3586589254 >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 0x7f98e80d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 -- 172.21.15.132:0/3586589254 >> v1:172.21.15.132:6789/0 conn(0x7f98e80c5110 legacy=0x7f98e815a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ede00640 1 -- 172.21.15.132:0/3586589254 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e80cb2c0 con 0x7f98e81517c0 2024-03-27T06:14:33.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.508+0000 7f98ed400640 1 --2- 172.21.15.132:0/3586589254 >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 0x7f98e80d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:33.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.510+0000 7f98ede00640 1 --2- 172.21.15.132:0/3586589254 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009870 tx=0x7f98dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d19ff2d6f1839ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.510+0000 7f98eca00640 1 -- 172.21.15.132:0/3586589254 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc034070 con 0x7f98e81517c0 2024-03-27T06:14:33.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.510+0000 7f98eca00640 1 -- 172.21.15.132:0/3586589254 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc030840 con 0x7f98e81517c0 2024-03-27T06:14:33.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.510+0000 7f98eca00640 1 -- 172.21.15.132:0/3586589254 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc030c10 con 0x7f98e81517c0 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 -- 172.21.15.132:0/3586589254 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 msgr2=0x7f98e80cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 --2- 172.21.15.132:0/3586589254 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009870 tx=0x7f98dc02b980 comp rx=0 tx=0).stop 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 -- 172.21.15.132:0/3586589254 shutdown_connections 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 --2- 172.21.15.132:0/3586589254 >> v2:172.21.15.132:3300/0 conn(0x7f98e80cc2b0 0x7f98e80d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 --2- 172.21.15.132:0/3586589254 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e81517c0 0x7f98e80cd1c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 -- 172.21.15.132:0/3586589254 >> 172.21.15.132:0/3586589254 conn(0x7f98e806dc80 msgr2=0x7f98e806e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:33.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.511+0000 7f98ef200640 1 -- 172.21.15.132:0/3586589254 shutdown_connections 2024-03-27T06:14:33.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.512+0000 7f98ef200640 1 -- 172.21.15.132:0/3586589254 wait complete. 2024-03-27T06:14:33.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.514+0000 7f98ef200640 1 Processor -- start 2024-03-27T06:14:33.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.514+0000 7f98ef200640 1 -- start start 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 0x7f98e80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 0x7f98e80cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98e81517c0 0x7f98e80cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80cd700 con 0x7f98e81517c0 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80c5660 con 0x7f98e80c5110 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ef200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f98e80cd880 con 0x7f98e80cc2b0 2024-03-27T06:14:33.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ed400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 0x7f98e80cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ede00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 0x7f98e80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ee800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98e81517c0 0x7f98e80cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ed400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 0x7f98e80cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34752/0 (socket says 172.21.15.132:34752) 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.516+0000 7f98ed400640 1 -- 172.21.15.132:0/272755279 learned_addr learned my addr 172.21.15.132:0/272755279 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 -- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 msgr2=0x7f98e80cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 0x7f98e80cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 -- 172.21.15.132:0/272755279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98e81517c0 msgr2=0x7f98e80cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98e81517c0 0x7f98e80cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 -- 172.21.15.132:0/272755279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e81f3af0 con 0x7f98e80c5110 2024-03-27T06:14:33.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ede00640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 0x7f98e80b4dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98dc033000 tx=0x7f98dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc049070 con 0x7f98e80c5110 2024-03-27T06:14:33.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc030640 con 0x7f98e80c5110 2024-03-27T06:14:33.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e81f3db0 con 0x7f98e80c5110 2024-03-27T06:14:33.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc034040 con 0x7f98e80c5110 2024-03-27T06:14:33.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.517+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e81f4370 con 0x7f98e80c5110 2024-03-27T06:14:33.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.519+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e806d650 con 0x7f98e80c5110 2024-03-27T06:14:33.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.519+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f98dc04c070 con 0x7f98e80c5110 2024-03-27T06:14:33.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.521+0000 7f98e2000640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 0x7f98cc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.521+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f98dc0ad400 con 0x7f98e80c5110 2024-03-27T06:14:33.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.521+0000 7f98ed400640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 0x7f98cc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.522+0000 7f98ed400640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 0x7f98cc06ea20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98e814dc20 tx=0x7f98d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.529+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f98e806d650 con 0x7f98e80c5110 2024-03-27T06:14:33.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.735+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f98e814bf40 con 0x7f98e80c5110 2024-03-27T06:14:33.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.736+0000 7f98e2000640 1 -- 172.21.15.132:0/272755279 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f98e814bf40 con 0x7f98e80c5110 2024-03-27T06:14:33.727 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:14:33.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 msgr2=0x7f98cc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 0x7f98cc06ea20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98e814dc20 tx=0x7f98d8008040 comp rx=0 tx=0).stop 2024-03-27T06:14:33.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 msgr2=0x7f98e80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 0x7f98e80b4dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98dc033000 tx=0x7f98dc0046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 shutdown_connections 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98e81517c0 0x7f98e80cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f98e80cc2b0 0x7f98e80cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f98cc06c560 0x7f98cc06ea20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 --2- 172.21.15.132:0/272755279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e80c5110 0x7f98e80b4dc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 >> 172.21.15.132:0/272755279 conn(0x7f98e806dc80 msgr2=0x7f98e806e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 shutdown_connections 2024-03-27T06:14:33.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.739+0000 7f98ef200640 1 -- 172.21.15.132:0/272755279 wait complete. 2024-03-27T06:14:33.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:14:33.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.978+0000 7f7dfee00640 1 Processor -- start 2024-03-27T06:14:33.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.979+0000 7f7dfee00640 1 -- start start 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7dfee00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7df8154f80 0x7f7df815b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7dfee00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7df815bb40 con 0x7f7df80c8bd0 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7dfee00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7df80ccd30 con 0x7f7df80cb2f0 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7dfee00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f7df8155360 con 0x7f7df8154f80 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7df7400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7df80cb2f0 0x7f7df80d0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39900/0 (socket says 172.21.15.132:39900) 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.980+0000 7f7df7400640 1 -- 172.21.15.132:0/2337458935 learned_addr learned my addr 172.21.15.132:0/2337458935 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df7e00640 1 --2- 172.21.15.132:0/2337458935 >> v2:172.21.15.132:3300/0 conn(0x7f7df8154f80 0x7f7df815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df6a00640 1 -- 172.21.15.132:0/2337458935 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795449919 0 0) 0x7f7df80ccd30 con 0x7f7df80cb2f0 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df6a00640 1 -- 172.21.15.132:0/2337458935 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ddc003520 con 0x7f7df80cb2f0 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df7e00640 1 -- 172.21.15.132:0/2337458935 >> v1:172.21.15.134:6789/0 conn(0x7f7df80cb2f0 legacy=0x7f7df80d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df7e00640 1 -- 172.21.15.132:0/2337458935 >> v1:172.21.15.132:6789/0 conn(0x7f7df80c8bd0 legacy=0x7f7df80c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.981+0000 7f7df7e00640 1 -- 172.21.15.132:0/2337458935 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df80ccfd0 con 0x7f7df8154f80 2024-03-27T06:14:33.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.982+0000 7f7df7e00640 1 --2- 172.21.15.132:0/2337458935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df8154f80 0x7f7df815b600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7de80097a0 tx=0x7f7de802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d360d90b2912c77d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7df6a00640 1 -- 172.21.15.132:0/2337458935 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8035020 con 0x7f7df8154f80 2024-03-27T06:14:33.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7df6a00640 1 -- 172.21.15.132:0/2337458935 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de8031830 con 0x7f7df8154f80 2024-03-27T06:14:33.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7df6a00640 1 -- 172.21.15.132:0/2337458935 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8031be0 con 0x7f7df8154f80 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 -- 172.21.15.132:0/2337458935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df8154f80 msgr2=0x7f7df815b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 --2- 172.21.15.132:0/2337458935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df8154f80 0x7f7df815b600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7de80097a0 tx=0x7f7de802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 -- 172.21.15.132:0/2337458935 shutdown_connections 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 --2- 172.21.15.132:0/2337458935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df8154f80 0x7f7df815b600 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 -- 172.21.15.132:0/2337458935 >> 172.21.15.132:0/2337458935 conn(0x7f7df806d9f0 msgr2=0x7f7df806de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:33.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 -- 172.21.15.132:0/2337458935 shutdown_connections 2024-03-27T06:14:33.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.983+0000 7f7dfee00640 1 -- 172.21.15.132:0/2337458935 wait complete. 2024-03-27T06:14:33.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.985+0000 7f7dfee00640 1 Processor -- start 2024-03-27T06:14:33.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.985+0000 7f7dfee00640 1 -- start start 2024-03-27T06:14:33.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 0x7f7df8159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7df80cd870 con 0x7f7df80c8bd0 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7df80cd590 con 0x7f7df8154f80 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7dfee00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7df80cc0a0 con 0x7f7df80cb2f0 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.986+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7dfce00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 0x7f7df8159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52840/0 (socket says 172.21.15.132:52840) 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 -- 172.21.15.132:0/500948405 learned_addr learned my addr 172.21.15.132:0/500948405 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7dfce00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50968/0 (socket says 172.21.15.132:50968) 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 -- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 msgr2=0x7f7df8159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 0x7f7df8159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 -- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 msgr2=0x7f7df81572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:33.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:33.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 -- 172.21.15.132:0/500948405 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df815e850 con 0x7f7df80c8bd0 2024-03-27T06:14:33.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7400640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 0x7f7df8159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:33.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7dfce00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.987+0000 7f7df7e00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7de8034040 tx=0x7f7de80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.988+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8035040 con 0x7f7df80c8bd0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.988+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de8004250 con 0x7f7df80c8bd0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.988+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de803a640 con 0x7f7df80c8bd0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.988+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df815eaf0 con 0x7f7df80c8bd0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.988+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81f39c0 con 0x7f7df80c8bd0 2024-03-27T06:14:33.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.989+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df8068950 con 0x7f7df80c8bd0 2024-03-27T06:14:33.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.990+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7de8038020 con 0x7f7df80c8bd0 2024-03-27T06:14:33.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.992+0000 7f7defe00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 0x7f7dd806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:33.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.992+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7de80ad520 con 0x7f7df80c8bd0 2024-03-27T06:14:33.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.992+0000 7f7df7400640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 0x7f7dd806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:33.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.993+0000 7f7df7400640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 0x7f7dd806ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7df80b2f50 tx=0x7f7de4007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:33.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:33.995+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f7de807b560 con 0x7f7df80c8bd0 2024-03-27T06:14:34.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.186+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7df80cbe80 con 0x7f7df80c8bd0 2024-03-27T06:14:34.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.187+0000 7f7defe00640 1 -- 172.21.15.132:0/500948405 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f7de807f860 con 0x7f7df80c8bd0 2024-03-27T06:14:34.178 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:14:34.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.189+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 msgr2=0x7f7dd806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.189+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 0x7f7dd806ea20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7df80b2f50 tx=0x7f7de4007d30 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.189+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 msgr2=0x7f7df81f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.189+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7de8034040 tx=0x7f7de80046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 shutdown_connections 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7dd806c560 0x7f7dd806ea20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7df8154f80 0x7f7df81572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7df80cb2f0 0x7f7df8159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 --2- 172.21.15.132:0/500948405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7df80c8bd0 0x7f7df81f0a60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 >> 172.21.15.132:0/500948405 conn(0x7f7df806d9f0 msgr2=0x7f7df81527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 shutdown_connections 2024-03-27T06:14:34.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.190+0000 7f7dfee00640 1 -- 172.21.15.132:0/500948405 wait complete. 2024-03-27T06:14:34.198 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-03-27T06:14:34.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:34.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.392+0000 7f3a7be00640 1 Processor -- start 2024-03-27T06:14:34.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.392+0000 7f3a7be00640 1 -- start start 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3a74154e00 0x7f3a7415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 0x7f3a740c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f3a7415ba80 con 0x7f3a74154e00 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3a7406ba60 con 0x7f3a7415bf30 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.394+0000 7f3a7be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7406bd40 con 0x7f3a7406c3d0 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a78a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 0x7f3a740c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3a74154e00 0x7f3a7415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34770/0 (socket says 172.21.15.132:34770) 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 -- 172.21.15.132:0/711844883 learned_addr learned my addr 172.21.15.132:0/711844883 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a78a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 0x7f3a740c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50978/0 (socket says 172.21.15.132:50978) 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 -- 172.21.15.132:0/711844883 >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 msgr2=0x7f3a740c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:34.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 --2- 172.21.15.132:0/711844883 >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 0x7f3a740c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 -- 172.21.15.132:0/711844883 >> v2:172.21.15.132:3300/0 conn(0x7f3a74154e00 msgr2=0x7f3a7415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:34.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 --2- 172.21.15.132:0/711844883 >> v2:172.21.15.132:3300/0 conn(0x7f3a74154e00 0x7f3a7415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.395+0000 7f3a79e00640 1 -- 172.21.15.132:0/711844883 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a740c8230 con 0x7f3a7406c3d0 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.396+0000 7f3a79e00640 1 --2- 172.21.15.132:0/711844883 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3a7408d6e0 tx=0x7f3a6c02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=509ab89e7933910 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a73e00640 1 -- 172.21.15.132:0/711844883 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c02ca80 con 0x7f3a7406c3d0 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a73e00640 1 -- 172.21.15.132:0/711844883 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a6c02cc20 con 0x7f3a7406c3d0 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a73e00640 1 -- 172.21.15.132:0/711844883 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0343f0 con 0x7f3a7406c3d0 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 -- 172.21.15.132:0/711844883 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 msgr2=0x7f3a740c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 --2- 172.21.15.132:0/711844883 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3a7408d6e0 tx=0x7f3a6c02bee0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 -- 172.21.15.132:0/711844883 shutdown_connections 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 --2- 172.21.15.132:0/711844883 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a7406c3d0 0x7f3a740c7550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 --2- 172.21.15.132:0/711844883 >> v2:172.21.15.134:3300/0 conn(0x7f3a7415bf30 0x7f3a740c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 --2- 172.21.15.132:0/711844883 >> v2:172.21.15.132:3300/0 conn(0x7f3a74154e00 0x7f3a7415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 -- 172.21.15.132:0/711844883 >> 172.21.15.132:0/711844883 conn(0x7f3a740d0150 msgr2=0x7f3a740d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:34.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 -- 172.21.15.132:0/711844883 shutdown_connections 2024-03-27T06:14:34.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.397+0000 7f3a7be00640 1 -- 172.21.15.132:0/711844883 wait complete. 2024-03-27T06:14:34.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.399+0000 7f3a7be00640 1 Processor -- start 2024-03-27T06:14:34.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.399+0000 7f3a7be00640 1 -- start start 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 0x7f3a741487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741551e0 con 0x7f3a7406c3d0 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a740c87f0 con 0x7f3a7415bf30 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a7be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3a740c9250 con 0x7f3a74154e00 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a78a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a79e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a79400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 0x7f3a741487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a78a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34772/0 (socket says 172.21.15.132:34772) 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a79e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50984/0 (socket says 172.21.15.132:50984) 2024-03-27T06:14:34.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.400+0000 7f3a78a00640 1 -- 172.21.15.132:0/1322951965 learned_addr learned my addr 172.21.15.132:0/1322951965 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:34.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 -- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 msgr2=0x7f3a74151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 -- 172.21.15.132:0/1322951965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 msgr2=0x7f3a741487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 0x7f3a741487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 -- 172.21.15.132:0/1322951965 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a740cb000 con 0x7f3a7415bf30 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a78a00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79400640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 0x7f3a741487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a79e00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a7406bec0 tx=0x7f3a6c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c042020 con 0x7f3a7415bf30 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a6c03e450 con 0x7f3a7415bf30 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a74148d30 con 0x7f3a7415bf30 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c009510 con 0x7f3a7415bf30 2024-03-27T06:14:34.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.401+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a740cb2a0 con 0x7f3a7415bf30 2024-03-27T06:14:34.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.402+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a74068840 con 0x7f3a7415bf30 2024-03-27T06:14:34.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.403+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3a6c045020 con 0x7f3a7415bf30 2024-03-27T06:14:34.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.405+0000 7f3a71600640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 0x7f3a5406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.405+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3a6c0ad9e0 con 0x7f3a7415bf30 2024-03-27T06:14:34.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.405+0000 7f3a79400640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 0x7f3a5406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.406+0000 7f3a79400640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 0x7f3a5406e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009f30 tx=0x7f3a68009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.413+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3a6c04c030 con 0x7f3a7415bf30 2024-03-27T06:14:34.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.680+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a740c3af0 con 0x7f3a7415bf30 2024-03-27T06:14:34.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.683+0000 7f3a71600640 1 -- 172.21.15.132:0/1322951965 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7f3a6c07b880 con 0x7f3a7415bf30 2024-03-27T06:14:34.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:34.675 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:14:34.676 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 msgr2=0x7f3a5406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 0x7f3a5406e9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009f30 tx=0x7f3a68009190 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 msgr2=0x7f3a741519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a7406bec0 tx=0x7f3a6c004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 shutdown_connections 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3a5406c530 0x7f3a5406e9f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7415bf30 0x7f3a741519c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3a74154e00 0x7f3a74151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 --2- 172.21.15.132:0/1322951965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a7406c3d0 0x7f3a741487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 >> 172.21.15.132:0/1322951965 conn(0x7f3a740d0150 msgr2=0x7f3a7415e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:34.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 shutdown_connections 2024-03-27T06:14:34.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.687+0000 7f3a7be00640 1 -- 172.21.15.132:0/1322951965 wait complete. 2024-03-27T06:14:34.703 INFO:tasks.mds_thrash:Ready to start thrashing 2024-03-27T06:14:34.703 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-03-27T06:14:34.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:34.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.890+0000 7fbb3f600640 1 Processor -- start 2024-03-27T06:14:34.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.890+0000 7fbb3f600640 1 -- start start 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3f600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3f600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbb38073880 0x7fbb38157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3f600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3806b520 con 0x7fbb381516f0 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3f600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fbb38072be0 con 0x7fbb38073880 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3f600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3806b7c0 con 0x7fbb38151ad0 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34780/0 (socket says 172.21.15.132:34780) 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3cc00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbb381516f0 0x7fbb380c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39910/0 (socket says 172.21.15.132:39910) 2024-03-27T06:14:34.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 -- 172.21.15.132:0/2330446269 learned_addr learned my addr 172.21.15.132:0/2330446269 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb3d600640 1 --2- 172.21.15.132:0/2330446269 >> v2:172.21.15.134:3300/0 conn(0x7fbb38073880 0x7fbb38157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 -- 172.21.15.132:0/2330446269 >> v2:172.21.15.134:3300/0 conn(0x7fbb38073880 msgr2=0x7fbb38157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 --2- 172.21.15.132:0/2330446269 >> v2:172.21.15.134:3300/0 conn(0x7fbb38073880 0x7fbb38157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 -- 172.21.15.132:0/2330446269 >> v1:172.21.15.134:6789/0 conn(0x7fbb381516f0 legacy=0x7fbb380c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.892+0000 7fbb37e00640 1 -- 172.21.15.132:0/2330446269 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3806baa0 con 0x7fbb38151ad0 2024-03-27T06:14:34.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.893+0000 7fbb37e00640 1 --2- 172.21.15.132:0/2330446269 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63cf7c9606bc73fc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.894+0000 7fbb37400640 1 -- 172.21.15.132:0/2330446269 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2402f840 con 0x7fbb38151ad0 2024-03-27T06:14:34.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.894+0000 7fbb37400640 1 -- 172.21.15.132:0/2330446269 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2402f9e0 con 0x7fbb38151ad0 2024-03-27T06:14:34.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.894+0000 7fbb37400640 1 -- 172.21.15.132:0/2330446269 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2402fdb0 con 0x7fbb38151ad0 2024-03-27T06:14:34.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.894+0000 7fbb3f600640 1 -- 172.21.15.132:0/2330446269 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 msgr2=0x7fbb38157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.894+0000 7fbb3f600640 1 --2- 172.21.15.132:0/2330446269 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2402ba20 comp rx=0 tx=0).stop 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 -- 172.21.15.132:0/2330446269 shutdown_connections 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 --2- 172.21.15.132:0/2330446269 >> v2:172.21.15.134:3300/0 conn(0x7fbb38073880 0x7fbb38157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 --2- 172.21.15.132:0/2330446269 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38151ad0 0x7fbb38157570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 -- 172.21.15.132:0/2330446269 >> 172.21.15.132:0/2330446269 conn(0x7fbb380cc060 msgr2=0x7fbb380cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 -- 172.21.15.132:0/2330446269 shutdown_connections 2024-03-27T06:14:34.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.895+0000 7fbb3f600640 1 -- 172.21.15.132:0/2330446269 wait complete. 2024-03-27T06:14:34.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.897+0000 7fbb3f600640 1 Processor -- start 2024-03-27T06:14:34.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.897+0000 7fbb3f600640 1 -- start start 2024-03-27T06:14:34.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 0x7fbb381543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 0x7fbb38156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb38151ad0 0x7fbb381457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3806bd80 con 0x7fbb38151ad0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38072870 con 0x7fbb381516f0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3f600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbb380c1080 con 0x7fbb38073880 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3cc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 0x7fbb381543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb37e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 0x7fbb38156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3cc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 0x7fbb381543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34790/0 (socket says 172.21.15.132:34790) 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.898+0000 7fbb3cc00640 1 -- 172.21.15.132:0/1313998531 learned_addr learned my addr 172.21.15.132:0/1313998531 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb3d600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb38151ad0 0x7fbb381457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 -- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 msgr2=0x7fbb381543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 0x7fbb381543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 -- 172.21.15.132:0/1313998531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb38151ad0 msgr2=0x7fbb381457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb38151ad0 0x7fbb381457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 -- 172.21.15.132:0/1313998531 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381f05f0 con 0x7fbb381516f0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb37e00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 0x7fbb38156360 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb24033000 tx=0x7fbb24004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24037bc0 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb24046070 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24041400 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.899+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3814bf10 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.900+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381efb10 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.900+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb38068840 con 0x7fbb381516f0 2024-03-27T06:14:34.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.901+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbb24040070 con 0x7fbb381516f0 2024-03-27T06:14:34.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.903+0000 7fbb34c00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 0x7fbb1806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:34.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.903+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb240adc50 con 0x7fbb381516f0 2024-03-27T06:14:34.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.903+0000 7fbb3cc00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 0x7fbb1806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:34.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.903+0000 7fbb3cc00640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 0x7fbb1806e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007fd0 tx=0x7fbb2c001740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:34.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:34.910+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fbb24033030 con 0x7fbb381516f0 2024-03-27T06:14:35.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.106+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb3814bcc0 con 0x7fbb1806c530 2024-03-27T06:14:35.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.114+0000 7fbb34c00640 1 -- 172.21.15.132:0/1313998531 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251989 (secure 0 0 0) 0x7fbb3814bcc0 con 0x7fbb1806c530 2024-03-27T06:14:35.109 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:35.110 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.118+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 msgr2=0x7fbb1806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.118+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 0x7fbb1806e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007fd0 tx=0x7fbb2c001740 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.118+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 msgr2=0x7fbb38156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.118+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 0x7fbb38156360 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbb24033000 tx=0x7fbb24004730 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 shutdown_connections 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb38151ad0 0x7fbb381457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbb1806c530 0x7fbb1806e9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb381516f0 0x7fbb38156360 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 --2- 172.21.15.132:0/1313998531 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbb38073880 0x7fbb381543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 >> 172.21.15.132:0/1313998531 conn(0x7fbb380cc060 msgr2=0x7fbb380714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:35.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 shutdown_connections 2024-03-27T06:14:35.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.119+0000 7fbb3f600640 1 -- 172.21.15.132:0/1313998531 wait complete. 2024-03-27T06:14:35.134 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-27T06:14:34.397027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595844,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":286,"num_write_kb":2456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218112,"kb_used_data":1984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756608,"statfs":{"total":773094113280,"available":772870766592,"internally_reserved":0,"allocated":2031616,"data_stored":1796528,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048058+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:07.048058+0000","last_peered":"2024-03-27T06:14:07.048058+0000","last_clean":"2024-03-27T06:14:07.048058+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:07.048058+0000","last_undegraded":"2024-03-27T06:14:07.048058+0000","last_fullsized":"2024-03-27T06:14:07.048058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708723+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:06.708723+0000","last_peered":"2024-03-27T06:14:06.708723+0000","last_clean":"2024-03-27T06:14:06.708723+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:06.708723+0000","last_undegraded":"2024-03-27T06:14:06.708723+0000","last_fullsized":"2024-03-27T06:14:06.708723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708489+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:06.708489+0000","last_peered":"2024-03-27T06:14:06.708489+0000","last_clean":"2024-03-27T06:14:06.708489+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:06.708489+0000","last_undegraded":"2024-03-27T06:14:06.708489+0000","last_fullsized":"2024-03-27T06:14:06.708489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462134+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:07.462134+0000","last_peered":"2024-03-27T06:14:07.462134+0000","last_clean":"2024-03-27T06:14:07.462134+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:07.462134+0000","last_undegraded":"2024-03-27T06:14:07.462134+0000","last_fullsized":"2024-03-27T06:14:07.462134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048224+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:07.048224+0000","last_peered":"2024-03-27T06:14:07.048224+0000","last_clean":"2024-03-27T06:14:07.048224+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:07.048224+0000","last_undegraded":"2024-03-27T06:14:07.048224+0000","last_fullsized":"2024-03-27T06:14:07.048224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462236+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:07.462236+0000","last_peered":"2024-03-27T06:14:07.462236+0000","last_clean":"2024-03-27T06:14:07.462236+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:07.462236+0000","last_undegraded":"2024-03-27T06:14:07.462236+0000","last_fullsized":"2024-03-27T06:14:07.462236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047358+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:07.047358+0000","last_peered":"2024-03-27T06:14:07.047358+0000","last_clean":"2024-03-27T06:14:07.047358+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:07.047358+0000","last_undegraded":"2024-03-27T06:14:07.047358+0000","last_fullsized":"2024-03-27T06:14:07.047358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708492+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:06.708492+0000","last_peered":"2024-03-27T06:14:06.708492+0000","last_clean":"2024-03-27T06:14:06.708492+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:06.708492+0000","last_undegraded":"2024-03-27T06:14:06.708492+0000","last_fullsized":"2024-03-27T06:14:06.708492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802323+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:06.802323+0000","last_peered":"2024-03-27T06:14:06.802323+0000","last_clean":"2024-03-27T06:14:06.802323+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:06.802323+0000","last_undegraded":"2024-03-27T06:14:06.802323+0000","last_fullsized":"2024-03-27T06:14:06.802323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048760+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:07.048760+0000","last_peered":"2024-03-27T06:14:07.048760+0000","last_clean":"2024-03-27T06:14:07.048760+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:07.048760+0000","last_undegraded":"2024-03-27T06:14:07.048760+0000","last_fullsized":"2024-03-27T06:14:07.048760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048755+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:07.048755+0000","last_peered":"2024-03-27T06:14:07.048755+0000","last_clean":"2024-03-27T06:14:07.048755+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:07.048755+0000","last_undegraded":"2024-03-27T06:14:07.048755+0000","last_fullsized":"2024-03-27T06:14:07.048755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047765+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:07.047765+0000","last_peered":"2024-03-27T06:14:07.047765+0000","last_clean":"2024-03-27T06:14:07.047765+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:07.047765+0000","last_undegraded":"2024-03-27T06:14:07.047765+0000","last_fullsized":"2024-03-27T06:14:07.047765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808956+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:07.808956+0000","last_peered":"2024-03-27T06:14:07.808956+0000","last_clean":"2024-03-27T06:14:07.808956+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:07.808956+0000","last_undegraded":"2024-03-27T06:14:07.808956+0000","last_fullsized":"2024-03-27T06:14:07.808956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808841+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:07.808841+0000","last_peered":"2024-03-27T06:14:07.808841+0000","last_clean":"2024-03-27T06:14:07.808841+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:07.808841+0000","last_undegraded":"2024-03-27T06:14:07.808841+0000","last_fullsized":"2024-03-27T06:14:07.808841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708767+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:06.708767+0000","last_peered":"2024-03-27T06:14:06.708767+0000","last_clean":"2024-03-27T06:14:06.708767+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:06.708767+0000","last_undegraded":"2024-03-27T06:14:06.708767+0000","last_fullsized":"2024-03-27T06:14:06.708767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048108+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:07.048108+0000","last_peered":"2024-03-27T06:14:07.048108+0000","last_clean":"2024-03-27T06:14:07.048108+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:07.048108+0000","last_undegraded":"2024-03-27T06:14:07.048108+0000","last_fullsized":"2024-03-27T06:14:07.048108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046295+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:07.046295+0000","last_peered":"2024-03-27T06:14:07.046295+0000","last_clean":"2024-03-27T06:14:07.046295+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:07.046295+0000","last_undegraded":"2024-03-27T06:14:07.046295+0000","last_fullsized":"2024-03-27T06:14:07.046295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801932+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:06.801932+0000","last_peered":"2024-03-27T06:14:06.801932+0000","last_clean":"2024-03-27T06:14:06.801932+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:06.801932+0000","last_undegraded":"2024-03-27T06:14:06.801932+0000","last_fullsized":"2024-03-27T06:14:06.801932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708558+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:06.708558+0000","last_peered":"2024-03-27T06:14:06.708558+0000","last_clean":"2024-03-27T06:14:06.708558+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:06.708558+0000","last_undegraded":"2024-03-27T06:14:06.708558+0000","last_fullsized":"2024-03-27T06:14:06.708558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708736+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:06.708736+0000","last_peered":"2024-03-27T06:14:06.708736+0000","last_clean":"2024-03-27T06:14:06.708736+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:06.708736+0000","last_undegraded":"2024-03-27T06:14:06.708736+0000","last_fullsized":"2024-03-27T06:14:06.708736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462807+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:07.462807+0000","last_peered":"2024-03-27T06:14:07.462807+0000","last_clean":"2024-03-27T06:14:07.462807+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:07.462807+0000","last_undegraded":"2024-03-27T06:14:07.462807+0000","last_fullsized":"2024-03-27T06:14:07.462807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809252+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:07.809252+0000","last_peered":"2024-03-27T06:14:07.809252+0000","last_clean":"2024-03-27T06:14:07.809252+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:07.809252+0000","last_undegraded":"2024-03-27T06:14:07.809252+0000","last_fullsized":"2024-03-27T06:14:07.809252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809134+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:07.809134+0000","last_peered":"2024-03-27T06:14:07.809134+0000","last_clean":"2024-03-27T06:14:07.809134+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:07.809134+0000","last_undegraded":"2024-03-27T06:14:07.809134+0000","last_fullsized":"2024-03-27T06:14:07.809134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708534+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:06.708534+0000","last_peered":"2024-03-27T06:14:06.708534+0000","last_clean":"2024-03-27T06:14:06.708534+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:06.708534+0000","last_undegraded":"2024-03-27T06:14:06.708534+0000","last_fullsized":"2024-03-27T06:14:06.708534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.461979+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:07.461979+0000","last_peered":"2024-03-27T06:14:07.461979+0000","last_clean":"2024-03-27T06:14:07.461979+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:07.461979+0000","last_undegraded":"2024-03-27T06:14:07.461979+0000","last_fullsized":"2024-03-27T06:14:07.461979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802104+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:06.802104+0000","last_peered":"2024-03-27T06:14:06.802104+0000","last_clean":"2024-03-27T06:14:06.802104+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:06.802104+0000","last_undegraded":"2024-03-27T06:14:06.802104+0000","last_fullsized":"2024-03-27T06:14:06.802104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.577932+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:13.577932+0000","last_peered":"2024-03-27T06:14:13.577932+0000","last_clean":"2024-03-27T06:14:13.577932+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:13.577932+0000","last_undegraded":"2024-03-27T06:14:13.577932+0000","last_fullsized":"2024-03-27T06:14:13.577932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046346+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:07.046346+0000","last_peered":"2024-03-27T06:14:07.046346+0000","last_clean":"2024-03-27T06:14:07.046346+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:07.046346+0000","last_undegraded":"2024-03-27T06:14:07.046346+0000","last_fullsized":"2024-03-27T06:14:07.046346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608838+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:18.608838+0000","last_peered":"2024-03-27T06:14:18.608838+0000","last_clean":"2024-03-27T06:14:18.608838+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:18.608838+0000","last_undegraded":"2024-03-27T06:14:18.608838+0000","last_fullsized":"2024-03-27T06:14:18.608838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608211+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:18.608211+0000","last_peered":"2024-03-27T06:14:18.608211+0000","last_clean":"2024-03-27T06:14:18.608211+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:18.608211+0000","last_undegraded":"2024-03-27T06:14:18.608211+0000","last_fullsized":"2024-03-27T06:14:18.608211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708759+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:06.708759+0000","last_peered":"2024-03-27T06:14:06.708759+0000","last_clean":"2024-03-27T06:14:06.708759+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:06.708759+0000","last_undegraded":"2024-03-27T06:14:06.708759+0000","last_fullsized":"2024-03-27T06:14:06.708759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431193+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:05.431193+0000","last_peered":"2024-03-27T06:14:05.431193+0000","last_clean":"2024-03-27T06:14:05.431193+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:05.431193+0000","last_undegraded":"2024-03-27T06:14:05.431193+0000","last_fullsized":"2024-03-27T06:14:05.431193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046377+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:07.046377+0000","last_peered":"2024-03-27T06:14:07.046377+0000","last_clean":"2024-03-27T06:14:07.046377+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:07.046377+0000","last_undegraded":"2024-03-27T06:14:07.046377+0000","last_fullsized":"2024-03-27T06:14:07.046377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.579962+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:13.579962+0000","last_peered":"2024-03-27T06:14:13.579962+0000","last_clean":"2024-03-27T06:14:13.579962+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:13.579962+0000","last_undegraded":"2024-03-27T06:14:13.579962+0000","last_fullsized":"2024-03-27T06:14:13.579962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462989+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:07.462989+0000","last_peered":"2024-03-27T06:14:07.462989+0000","last_clean":"2024-03-27T06:14:07.462989+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:07.462989+0000","last_undegraded":"2024-03-27T06:14:07.462989+0000","last_fullsized":"2024-03-27T06:14:07.462989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582879+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:13.582879+0000","last_peered":"2024-03-27T06:14:13.582879+0000","last_clean":"2024-03-27T06:14:13.582879+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:13.582879+0000","last_undegraded":"2024-03-27T06:14:13.582879+0000","last_fullsized":"2024-03-27T06:14:13.582879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708885+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:06.708885+0000","last_peered":"2024-03-27T06:14:06.708885+0000","last_clean":"2024-03-27T06:14:06.708885+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:06.708885+0000","last_undegraded":"2024-03-27T06:14:06.708885+0000","last_fullsized":"2024-03-27T06:14:06.708885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.516309+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:08.516309+0000","last_peered":"2024-03-27T06:14:08.516309+0000","last_clean":"2024-03-27T06:14:08.516309+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:08.516309+0000","last_undegraded":"2024-03-27T06:14:08.516309+0000","last_fullsized":"2024-03-27T06:14:08.516309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:29.461255+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:29.461255+0000","last_peered":"2024-03-27T06:14:29.461255+0000","last_clean":"2024-03-27T06:14:29.461255+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:29.461255+0000","last_undegraded":"2024-03-27T06:14:29.461255+0000","last_fullsized":"2024-03-27T06:14:29.461255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.049066+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:07.049066+0000","last_peered":"2024-03-27T06:14:07.049066+0000","last_clean":"2024-03-27T06:14:07.049066+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:07.049066+0000","last_undegraded":"2024-03-27T06:14:07.049066+0000","last_fullsized":"2024-03-27T06:14:07.049066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608165+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:18.608165+0000","last_peered":"2024-03-27T06:14:18.608165+0000","last_clean":"2024-03-27T06:14:18.608165+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:18.608165+0000","last_undegraded":"2024-03-27T06:14:18.608165+0000","last_fullsized":"2024-03-27T06:14:18.608165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808902+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:07.808902+0000","last_peered":"2024-03-27T06:14:07.808902+0000","last_clean":"2024-03-27T06:14:07.808902+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:07.808902+0000","last_undegraded":"2024-03-27T06:14:07.808902+0000","last_fullsized":"2024-03-27T06:14:07.808902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581161+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:13.581161+0000","last_peered":"2024-03-27T06:14:13.581161+0000","last_clean":"2024-03-27T06:14:13.581161+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:13.581161+0000","last_undegraded":"2024-03-27T06:14:13.581161+0000","last_fullsized":"2024-03-27T06:14:13.581161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613984+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:18.613984+0000","last_peered":"2024-03-27T06:14:18.613984+0000","last_clean":"2024-03-27T06:14:18.613984+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:18.613984+0000","last_undegraded":"2024-03-27T06:14:18.613984+0000","last_fullsized":"2024-03-27T06:14:18.613984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802187+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:06.802187+0000","last_peered":"2024-03-27T06:14:06.802187+0000","last_clean":"2024-03-27T06:14:06.802187+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:06.802187+0000","last_undegraded":"2024-03-27T06:14:06.802187+0000","last_fullsized":"2024-03-27T06:14:06.802187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.606607+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:18.606607+0000","last_peered":"2024-03-27T06:14:18.606607+0000","last_clean":"2024-03-27T06:14:18.606607+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:18.606607+0000","last_undegraded":"2024-03-27T06:14:18.606607+0000","last_fullsized":"2024-03-27T06:14:18.606607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.524195+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:08.524195+0000","last_peered":"2024-03-27T06:14:08.524195+0000","last_clean":"2024-03-27T06:14:08.524195+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:08.524195+0000","last_undegraded":"2024-03-27T06:14:08.524195+0000","last_fullsized":"2024-03-27T06:14:08.524195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.463061+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:07.463061+0000","last_peered":"2024-03-27T06:14:07.463061+0000","last_clean":"2024-03-27T06:14:07.463061+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:07.463061+0000","last_undegraded":"2024-03-27T06:14:07.463061+0000","last_fullsized":"2024-03-27T06:14:07.463061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.517650+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:08.517650+0000","last_peered":"2024-03-27T06:14:08.517650+0000","last_clean":"2024-03-27T06:14:08.517650+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:08.517650+0000","last_undegraded":"2024-03-27T06:14:08.517650+0000","last_fullsized":"2024-03-27T06:14:08.517650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047986+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:07.047986+0000","last_peered":"2024-03-27T06:14:07.047986+0000","last_clean":"2024-03-27T06:14:07.047986+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:07.047986+0000","last_undegraded":"2024-03-27T06:14:07.047986+0000","last_fullsized":"2024-03-27T06:14:07.047986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708579+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:06.708579+0000","last_peered":"2024-03-27T06:14:06.708579+0000","last_clean":"2024-03-27T06:14:06.708579+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:06.708579+0000","last_undegraded":"2024-03-27T06:14:06.708579+0000","last_fullsized":"2024-03-27T06:14:06.708579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048156+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:07.048156+0000","last_peered":"2024-03-27T06:14:07.048156+0000","last_clean":"2024-03-27T06:14:07.048156+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:07.048156+0000","last_undegraded":"2024-03-27T06:14:07.048156+0000","last_fullsized":"2024-03-27T06:14:07.048156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608234+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:18.608234+0000","last_peered":"2024-03-27T06:14:18.608234+0000","last_clean":"2024-03-27T06:14:18.608234+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:18.608234+0000","last_undegraded":"2024-03-27T06:14:18.608234+0000","last_fullsized":"2024-03-27T06:14:18.608234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801767+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:06.801767+0000","last_peered":"2024-03-27T06:14:06.801767+0000","last_clean":"2024-03-27T06:14:06.801767+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:06.801767+0000","last_undegraded":"2024-03-27T06:14:06.801767+0000","last_fullsized":"2024-03-27T06:14:06.801767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584731+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:13.584731+0000","last_peered":"2024-03-27T06:14:13.584731+0000","last_clean":"2024-03-27T06:14:13.584731+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:13.584731+0000","last_undegraded":"2024-03-27T06:14:13.584731+0000","last_fullsized":"2024-03-27T06:14:13.584731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608966+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:18.608966+0000","last_peered":"2024-03-27T06:14:18.608966+0000","last_clean":"2024-03-27T06:14:18.608966+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:18.608966+0000","last_undegraded":"2024-03-27T06:14:18.608966+0000","last_fullsized":"2024-03-27T06:14:18.608966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808793+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:07.808793+0000","last_peered":"2024-03-27T06:14:07.808793+0000","last_clean":"2024-03-27T06:14:07.808793+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:07.808793+0000","last_undegraded":"2024-03-27T06:14:07.808793+0000","last_fullsized":"2024-03-27T06:14:07.808793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.605510+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:18.605510+0000","last_peered":"2024-03-27T06:14:18.605510+0000","last_clean":"2024-03-27T06:14:18.605510+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:18.605510+0000","last_undegraded":"2024-03-27T06:14:18.605510+0000","last_fullsized":"2024-03-27T06:14:18.605510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708423+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:06.708423+0000","last_peered":"2024-03-27T06:14:06.708423+0000","last_clean":"2024-03-27T06:14:06.708423+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:06.708423+0000","last_undegraded":"2024-03-27T06:14:06.708423+0000","last_fullsized":"2024-03-27T06:14:06.708423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431429+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:05.431429+0000","last_peered":"2024-03-27T06:14:05.431429+0000","last_clean":"2024-03-27T06:14:05.431429+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:05.431429+0000","last_undegraded":"2024-03-27T06:14:05.431429+0000","last_fullsized":"2024-03-27T06:14:05.431429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048444+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:07.048444+0000","last_peered":"2024-03-27T06:14:07.048444+0000","last_clean":"2024-03-27T06:14:07.048444+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:07.048444+0000","last_undegraded":"2024-03-27T06:14:07.048444+0000","last_fullsized":"2024-03-27T06:14:07.048444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708415+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:06.708415+0000","last_peered":"2024-03-27T06:14:06.708415+0000","last_clean":"2024-03-27T06:14:06.708415+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:06.708415+0000","last_undegraded":"2024-03-27T06:14:06.708415+0000","last_fullsized":"2024-03-27T06:14:06.708415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462059+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:07.462059+0000","last_peered":"2024-03-27T06:14:07.462059+0000","last_clean":"2024-03-27T06:14:07.462059+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:07.462059+0000","last_undegraded":"2024-03-27T06:14:07.462059+0000","last_fullsized":"2024-03-27T06:14:07.462059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431591+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:05.431591+0000","last_peered":"2024-03-27T06:14:05.431591+0000","last_clean":"2024-03-27T06:14:05.431591+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:05.431591+0000","last_undegraded":"2024-03-27T06:14:05.431591+0000","last_fullsized":"2024-03-27T06:14:05.431591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809116+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:07.809116+0000","last_peered":"2024-03-27T06:14:07.809116+0000","last_clean":"2024-03-27T06:14:07.809116+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:07.809116+0000","last_undegraded":"2024-03-27T06:14:07.809116+0000","last_fullsized":"2024-03-27T06:14:07.809116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613716+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:18.613716+0000","last_peered":"2024-03-27T06:14:18.613716+0000","last_clean":"2024-03-27T06:14:18.613716+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:18.613716+0000","last_undegraded":"2024-03-27T06:14:18.613716+0000","last_fullsized":"2024-03-27T06:14:18.613716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046680+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:07.046680+0000","last_peered":"2024-03-27T06:14:07.046680+0000","last_clean":"2024-03-27T06:14:07.046680+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:07.046680+0000","last_undegraded":"2024-03-27T06:14:07.046680+0000","last_fullsized":"2024-03-27T06:14:07.046680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581236+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:13.581236+0000","last_peered":"2024-03-27T06:14:13.581236+0000","last_clean":"2024-03-27T06:14:13.581236+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:13.581236+0000","last_undegraded":"2024-03-27T06:14:13.581236+0000","last_fullsized":"2024-03-27T06:14:13.581236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046583+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:07.046583+0000","last_peered":"2024-03-27T06:14:07.046583+0000","last_clean":"2024-03-27T06:14:07.046583+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:07.046583+0000","last_undegraded":"2024-03-27T06:14:07.046583+0000","last_fullsized":"2024-03-27T06:14:07.046583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584347+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:13.584347+0000","last_peered":"2024-03-27T06:14:13.584347+0000","last_clean":"2024-03-27T06:14:13.584347+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:13.584347+0000","last_undegraded":"2024-03-27T06:14:13.584347+0000","last_fullsized":"2024-03-27T06:14:13.584347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582654+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:13.582654+0000","last_peered":"2024-03-27T06:14:13.582654+0000","last_clean":"2024-03-27T06:14:13.582654+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:13.582654+0000","last_undegraded":"2024-03-27T06:14:13.582654+0000","last_fullsized":"2024-03-27T06:14:13.582654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809031+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:07.809031+0000","last_peered":"2024-03-27T06:14:07.809031+0000","last_clean":"2024-03-27T06:14:07.809031+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:07.809031+0000","last_undegraded":"2024-03-27T06:14:07.809031+0000","last_fullsized":"2024-03-27T06:14:07.809031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.580898+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:13.580898+0000","last_peered":"2024-03-27T06:14:13.580898+0000","last_clean":"2024-03-27T06:14:13.580898+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:13.580898+0000","last_undegraded":"2024-03-27T06:14:13.580898+0000","last_fullsized":"2024-03-27T06:14:13.580898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525809+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:08.525809+0000","last_peered":"2024-03-27T06:14:08.525809+0000","last_clean":"2024-03-27T06:14:08.525809+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:08.525809+0000","last_undegraded":"2024-03-27T06:14:08.525809+0000","last_fullsized":"2024-03-27T06:14:08.525809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802159+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:06.802159+0000","last_peered":"2024-03-27T06:14:06.802159+0000","last_clean":"2024-03-27T06:14:06.802159+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:06.802159+0000","last_undegraded":"2024-03-27T06:14:06.802159+0000","last_fullsized":"2024-03-27T06:14:06.802159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613242+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:18.613242+0000","last_peered":"2024-03-27T06:14:18.613242+0000","last_clean":"2024-03-27T06:14:18.613242+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:18.613242+0000","last_undegraded":"2024-03-27T06:14:18.613242+0000","last_fullsized":"2024-03-27T06:14:18.613242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708697+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:06.708697+0000","last_peered":"2024-03-27T06:14:06.708697+0000","last_clean":"2024-03-27T06:14:06.708697+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:06.708697+0000","last_undegraded":"2024-03-27T06:14:06.708697+0000","last_fullsized":"2024-03-27T06:14:06.708697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.515193+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:08.515193+0000","last_peered":"2024-03-27T06:14:08.515193+0000","last_clean":"2024-03-27T06:14:08.515193+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:08.515193+0000","last_undegraded":"2024-03-27T06:14:08.515193+0000","last_fullsized":"2024-03-27T06:14:08.515193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708688+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:06.708688+0000","last_peered":"2024-03-27T06:14:06.708688+0000","last_clean":"2024-03-27T06:14:06.708688+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:06.708688+0000","last_undegraded":"2024-03-27T06:14:06.708688+0000","last_fullsized":"2024-03-27T06:14:06.708688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613729+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:18.613729+0000","last_peered":"2024-03-27T06:14:18.613729+0000","last_clean":"2024-03-27T06:14:18.613729+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:18.613729+0000","last_undegraded":"2024-03-27T06:14:18.613729+0000","last_fullsized":"2024-03-27T06:14:18.613729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5476,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":10952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":666777,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":79039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666133,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:35.139 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:35.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.386+0000 7fed9a800640 1 Processor -- start 2024-03-27T06:14:35.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.387+0000 7fed9a800640 1 -- start start 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.388+0000 7fed9a800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 0x7fed940cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.388+0000 7fed9a800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 0x7fed9415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.388+0000 7fed9a800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed9415de10 con 0x7fed940cb870 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.388+0000 7fed9a800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fed940cada0 con 0x7fed9415adb0 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.388+0000 7fed9a800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fed9415b720 con 0x7fed941549a0 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed98a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fed940cb870 0x7fed9406b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:55806/0 (socket says 172.21.15.132:55806) 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed98a00640 1 -- 172.21.15.132:0/2725057527 learned_addr learned my addr 172.21.15.132:0/2725057527 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99400640 1 --2- 172.21.15.132:0/2725057527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 0x7fed940cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99e00640 1 --2- 172.21.15.132:0/2725057527 >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 0x7fed9415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed93e00640 1 -- 172.21.15.132:0/2725057527 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2838803912 0 0) 0x7fed9415de10 con 0x7fed940cb870 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed93e00640 1 -- 172.21.15.132:0/2725057527 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed7c003520 con 0x7fed940cb870 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99400640 1 -- 172.21.15.132:0/2725057527 >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 msgr2=0x7fed9415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99400640 1 --2- 172.21.15.132:0/2725057527 >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 0x7fed9415b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99400640 1 -- 172.21.15.132:0/2725057527 >> v1:172.21.15.132:6789/0 conn(0x7fed940cb870 legacy=0x7fed9406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99400640 1 -- 172.21.15.132:0/2725057527 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9415e0b0 con 0x7fed941549a0 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.389+0000 7fed99e00640 1 --2- 172.21.15.132:0/2725057527 >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 0x7fed9415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:35.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.391+0000 7fed99400640 1 --2- 172.21.15.132:0/2725057527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 0x7fed940cd6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed880097a0 tx=0x7fed8802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a518dc54f1f1e61c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.391+0000 7fed93e00640 1 -- 172.21.15.132:0/2725057527 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88034070 con 0x7fed941549a0 2024-03-27T06:14:35.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.391+0000 7fed93e00640 1 -- 172.21.15.132:0/2725057527 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed88030800 con 0x7fed941549a0 2024-03-27T06:14:35.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.391+0000 7fed93e00640 1 -- 172.21.15.132:0/2725057527 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88030bd0 con 0x7fed941549a0 2024-03-27T06:14:35.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 -- 172.21.15.132:0/2725057527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 msgr2=0x7fed940cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 --2- 172.21.15.132:0/2725057527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 0x7fed940cd6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed880097a0 tx=0x7fed8802b940 comp rx=0 tx=0).stop 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 -- 172.21.15.132:0/2725057527 shutdown_connections 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 --2- 172.21.15.132:0/2725057527 >> v2:172.21.15.134:3300/0 conn(0x7fed9415adb0 0x7fed9415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 --2- 172.21.15.132:0/2725057527 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed941549a0 0x7fed940cd6a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 -- 172.21.15.132:0/2725057527 >> 172.21.15.132:0/2725057527 conn(0x7fed9406d8d0 msgr2=0x7fed9406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 -- 172.21.15.132:0/2725057527 shutdown_connections 2024-03-27T06:14:35.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.392+0000 7fed9a800640 1 -- 172.21.15.132:0/2725057527 wait complete. 2024-03-27T06:14:35.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.393+0000 7fed9a800640 1 Processor -- start 2024-03-27T06:14:35.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 -- start start 2024-03-27T06:14:35.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed941549a0 0x7fed94146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 0x7fed941f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed9415e950 con 0x7fed9415adb0 2024-03-27T06:14:35.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fed9415e670 con 0x7fed941549a0 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.394+0000 7fed9a800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fed9415ec30 con 0x7fed940cb870 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34810/0 (socket says 172.21.15.132:34810) 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 -- 172.21.15.132:0/3005614263 learned_addr learned my addr 172.21.15.132:0/3005614263 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed98a00640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed941549a0 0x7fed94146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99e00640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 0x7fed941f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 -- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed941549a0 msgr2=0x7fed94146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed941549a0 0x7fed94146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 -- 172.21.15.132:0/3005614263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 msgr2=0x7fed941f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 0x7fed941f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 -- 172.21.15.132:0/3005614263 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed941f3500 con 0x7fed940cb870 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99e00640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 0x7fed941f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed99400640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed88009890 tx=0x7fed880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.395+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88034040 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.396+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed88030e30 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.396+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed941478e0 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.396+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88039d40 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.396+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed94147e80 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.397+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed941f39b0 con 0x7fed940cb870 2024-03-27T06:14:35.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.398+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fed88037070 con 0x7fed940cb870 2024-03-27T06:14:35.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.398+0000 7fed91600640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 0x7fed7406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.399+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fed880ad690 con 0x7fed940cb870 2024-03-27T06:14:35.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.399+0000 7fed98a00640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 0x7fed7406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.390 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.399+0000 7fed98a00640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 0x7fed7406e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed9414ae80 tx=0x7fed80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.403+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fed8807bcb0 con 0x7fed940cb870 2024-03-27T06:14:35.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.597+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fed940686c0 con 0x7fed7406c530 2024-03-27T06:14:35.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.610+0000 7fed91600640 1 -- 172.21.15.132:0/3005614263 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251989 (secure 0 0 0) 0x7fed940686c0 con 0x7fed7406c530 2024-03-27T06:14:35.604 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:35.607 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:35.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 msgr2=0x7fed7406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 0x7fed7406e9f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed9414ae80 tx=0x7fed80009040 comp rx=0 tx=0).stop 2024-03-27T06:14:35.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 msgr2=0x7fed9414a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed88009890 tx=0x7fed880046c0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 shutdown_connections 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed9415adb0 0x7fed941f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fed7406c530 0x7fed7406e9f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed941549a0 0x7fed94146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 --2- 172.21.15.132:0/3005614263 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fed940cb870 0x7fed9414a180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 >> 172.21.15.132:0/3005614263 conn(0x7fed9406d8d0 msgr2=0x7fed9406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 shutdown_connections 2024-03-27T06:14:35.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.616+0000 7fed9a800640 1 -- 172.21.15.132:0/3005614263 wait complete. 2024-03-27T06:14:35.623 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-27T06:14:34.397027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595844,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":286,"num_write_kb":2456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218112,"kb_used_data":1984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756608,"statfs":{"total":773094113280,"available":772870766592,"internally_reserved":0,"allocated":2031616,"data_stored":1796528,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048058+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:07.048058+0000","last_peered":"2024-03-27T06:14:07.048058+0000","last_clean":"2024-03-27T06:14:07.048058+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:07.048058+0000","last_undegraded":"2024-03-27T06:14:07.048058+0000","last_fullsized":"2024-03-27T06:14:07.048058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708723+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:06.708723+0000","last_peered":"2024-03-27T06:14:06.708723+0000","last_clean":"2024-03-27T06:14:06.708723+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:06.708723+0000","last_undegraded":"2024-03-27T06:14:06.708723+0000","last_fullsized":"2024-03-27T06:14:06.708723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708489+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:06.708489+0000","last_peered":"2024-03-27T06:14:06.708489+0000","last_clean":"2024-03-27T06:14:06.708489+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:06.708489+0000","last_undegraded":"2024-03-27T06:14:06.708489+0000","last_fullsized":"2024-03-27T06:14:06.708489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462134+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:07.462134+0000","last_peered":"2024-03-27T06:14:07.462134+0000","last_clean":"2024-03-27T06:14:07.462134+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:07.462134+0000","last_undegraded":"2024-03-27T06:14:07.462134+0000","last_fullsized":"2024-03-27T06:14:07.462134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048224+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:07.048224+0000","last_peered":"2024-03-27T06:14:07.048224+0000","last_clean":"2024-03-27T06:14:07.048224+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:07.048224+0000","last_undegraded":"2024-03-27T06:14:07.048224+0000","last_fullsized":"2024-03-27T06:14:07.048224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462236+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:07.462236+0000","last_peered":"2024-03-27T06:14:07.462236+0000","last_clean":"2024-03-27T06:14:07.462236+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:07.462236+0000","last_undegraded":"2024-03-27T06:14:07.462236+0000","last_fullsized":"2024-03-27T06:14:07.462236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047358+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:07.047358+0000","last_peered":"2024-03-27T06:14:07.047358+0000","last_clean":"2024-03-27T06:14:07.047358+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:07.047358+0000","last_undegraded":"2024-03-27T06:14:07.047358+0000","last_fullsized":"2024-03-27T06:14:07.047358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708492+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:06.708492+0000","last_peered":"2024-03-27T06:14:06.708492+0000","last_clean":"2024-03-27T06:14:06.708492+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:06.708492+0000","last_undegraded":"2024-03-27T06:14:06.708492+0000","last_fullsized":"2024-03-27T06:14:06.708492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802323+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:06.802323+0000","last_peered":"2024-03-27T06:14:06.802323+0000","last_clean":"2024-03-27T06:14:06.802323+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:06.802323+0000","last_undegraded":"2024-03-27T06:14:06.802323+0000","last_fullsized":"2024-03-27T06:14:06.802323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048760+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:07.048760+0000","last_peered":"2024-03-27T06:14:07.048760+0000","last_clean":"2024-03-27T06:14:07.048760+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:07.048760+0000","last_undegraded":"2024-03-27T06:14:07.048760+0000","last_fullsized":"2024-03-27T06:14:07.048760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048755+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:07.048755+0000","last_peered":"2024-03-27T06:14:07.048755+0000","last_clean":"2024-03-27T06:14:07.048755+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:07.048755+0000","last_undegraded":"2024-03-27T06:14:07.048755+0000","last_fullsized":"2024-03-27T06:14:07.048755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047765+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:07.047765+0000","last_peered":"2024-03-27T06:14:07.047765+0000","last_clean":"2024-03-27T06:14:07.047765+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:07.047765+0000","last_undegraded":"2024-03-27T06:14:07.047765+0000","last_fullsized":"2024-03-27T06:14:07.047765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808956+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:07.808956+0000","last_peered":"2024-03-27T06:14:07.808956+0000","last_clean":"2024-03-27T06:14:07.808956+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:07.808956+0000","last_undegraded":"2024-03-27T06:14:07.808956+0000","last_fullsized":"2024-03-27T06:14:07.808956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808841+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:07.808841+0000","last_peered":"2024-03-27T06:14:07.808841+0000","last_clean":"2024-03-27T06:14:07.808841+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:07.808841+0000","last_undegraded":"2024-03-27T06:14:07.808841+0000","last_fullsized":"2024-03-27T06:14:07.808841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708767+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:06.708767+0000","last_peered":"2024-03-27T06:14:06.708767+0000","last_clean":"2024-03-27T06:14:06.708767+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:06.708767+0000","last_undegraded":"2024-03-27T06:14:06.708767+0000","last_fullsized":"2024-03-27T06:14:06.708767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048108+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:07.048108+0000","last_peered":"2024-03-27T06:14:07.048108+0000","last_clean":"2024-03-27T06:14:07.048108+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:07.048108+0000","last_undegraded":"2024-03-27T06:14:07.048108+0000","last_fullsized":"2024-03-27T06:14:07.048108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046295+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:07.046295+0000","last_peered":"2024-03-27T06:14:07.046295+0000","last_clean":"2024-03-27T06:14:07.046295+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:07.046295+0000","last_undegraded":"2024-03-27T06:14:07.046295+0000","last_fullsized":"2024-03-27T06:14:07.046295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801932+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:06.801932+0000","last_peered":"2024-03-27T06:14:06.801932+0000","last_clean":"2024-03-27T06:14:06.801932+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:06.801932+0000","last_undegraded":"2024-03-27T06:14:06.801932+0000","last_fullsized":"2024-03-27T06:14:06.801932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708558+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:06.708558+0000","last_peered":"2024-03-27T06:14:06.708558+0000","last_clean":"2024-03-27T06:14:06.708558+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:06.708558+0000","last_undegraded":"2024-03-27T06:14:06.708558+0000","last_fullsized":"2024-03-27T06:14:06.708558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708736+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:06.708736+0000","last_peered":"2024-03-27T06:14:06.708736+0000","last_clean":"2024-03-27T06:14:06.708736+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:06.708736+0000","last_undegraded":"2024-03-27T06:14:06.708736+0000","last_fullsized":"2024-03-27T06:14:06.708736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462807+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:07.462807+0000","last_peered":"2024-03-27T06:14:07.462807+0000","last_clean":"2024-03-27T06:14:07.462807+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:07.462807+0000","last_undegraded":"2024-03-27T06:14:07.462807+0000","last_fullsized":"2024-03-27T06:14:07.462807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809252+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:07.809252+0000","last_peered":"2024-03-27T06:14:07.809252+0000","last_clean":"2024-03-27T06:14:07.809252+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:07.809252+0000","last_undegraded":"2024-03-27T06:14:07.809252+0000","last_fullsized":"2024-03-27T06:14:07.809252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809134+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:07.809134+0000","last_peered":"2024-03-27T06:14:07.809134+0000","last_clean":"2024-03-27T06:14:07.809134+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:07.809134+0000","last_undegraded":"2024-03-27T06:14:07.809134+0000","last_fullsized":"2024-03-27T06:14:07.809134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708534+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:06.708534+0000","last_peered":"2024-03-27T06:14:06.708534+0000","last_clean":"2024-03-27T06:14:06.708534+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:06.708534+0000","last_undegraded":"2024-03-27T06:14:06.708534+0000","last_fullsized":"2024-03-27T06:14:06.708534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.461979+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:07.461979+0000","last_peered":"2024-03-27T06:14:07.461979+0000","last_clean":"2024-03-27T06:14:07.461979+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:07.461979+0000","last_undegraded":"2024-03-27T06:14:07.461979+0000","last_fullsized":"2024-03-27T06:14:07.461979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802104+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:06.802104+0000","last_peered":"2024-03-27T06:14:06.802104+0000","last_clean":"2024-03-27T06:14:06.802104+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:06.802104+0000","last_undegraded":"2024-03-27T06:14:06.802104+0000","last_fullsized":"2024-03-27T06:14:06.802104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.577932+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:13.577932+0000","last_peered":"2024-03-27T06:14:13.577932+0000","last_clean":"2024-03-27T06:14:13.577932+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:13.577932+0000","last_undegraded":"2024-03-27T06:14:13.577932+0000","last_fullsized":"2024-03-27T06:14:13.577932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046346+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:07.046346+0000","last_peered":"2024-03-27T06:14:07.046346+0000","last_clean":"2024-03-27T06:14:07.046346+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:07.046346+0000","last_undegraded":"2024-03-27T06:14:07.046346+0000","last_fullsized":"2024-03-27T06:14:07.046346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608838+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:18.608838+0000","last_peered":"2024-03-27T06:14:18.608838+0000","last_clean":"2024-03-27T06:14:18.608838+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:18.608838+0000","last_undegraded":"2024-03-27T06:14:18.608838+0000","last_fullsized":"2024-03-27T06:14:18.608838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608211+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:18.608211+0000","last_peered":"2024-03-27T06:14:18.608211+0000","last_clean":"2024-03-27T06:14:18.608211+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:18.608211+0000","last_undegraded":"2024-03-27T06:14:18.608211+0000","last_fullsized":"2024-03-27T06:14:18.608211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708759+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:06.708759+0000","last_peered":"2024-03-27T06:14:06.708759+0000","last_clean":"2024-03-27T06:14:06.708759+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:06.708759+0000","last_undegraded":"2024-03-27T06:14:06.708759+0000","last_fullsized":"2024-03-27T06:14:06.708759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431193+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:05.431193+0000","last_peered":"2024-03-27T06:14:05.431193+0000","last_clean":"2024-03-27T06:14:05.431193+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:05.431193+0000","last_undegraded":"2024-03-27T06:14:05.431193+0000","last_fullsized":"2024-03-27T06:14:05.431193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046377+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:07.046377+0000","last_peered":"2024-03-27T06:14:07.046377+0000","last_clean":"2024-03-27T06:14:07.046377+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:07.046377+0000","last_undegraded":"2024-03-27T06:14:07.046377+0000","last_fullsized":"2024-03-27T06:14:07.046377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.579962+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:13.579962+0000","last_peered":"2024-03-27T06:14:13.579962+0000","last_clean":"2024-03-27T06:14:13.579962+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:13.579962+0000","last_undegraded":"2024-03-27T06:14:13.579962+0000","last_fullsized":"2024-03-27T06:14:13.579962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462989+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:07.462989+0000","last_peered":"2024-03-27T06:14:07.462989+0000","last_clean":"2024-03-27T06:14:07.462989+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:07.462989+0000","last_undegraded":"2024-03-27T06:14:07.462989+0000","last_fullsized":"2024-03-27T06:14:07.462989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582879+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:13.582879+0000","last_peered":"2024-03-27T06:14:13.582879+0000","last_clean":"2024-03-27T06:14:13.582879+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:13.582879+0000","last_undegraded":"2024-03-27T06:14:13.582879+0000","last_fullsized":"2024-03-27T06:14:13.582879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708885+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:06.708885+0000","last_peered":"2024-03-27T06:14:06.708885+0000","last_clean":"2024-03-27T06:14:06.708885+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:06.708885+0000","last_undegraded":"2024-03-27T06:14:06.708885+0000","last_fullsized":"2024-03-27T06:14:06.708885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.516309+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:08.516309+0000","last_peered":"2024-03-27T06:14:08.516309+0000","last_clean":"2024-03-27T06:14:08.516309+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:08.516309+0000","last_undegraded":"2024-03-27T06:14:08.516309+0000","last_fullsized":"2024-03-27T06:14:08.516309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:29.461255+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:29.461255+0000","last_peered":"2024-03-27T06:14:29.461255+0000","last_clean":"2024-03-27T06:14:29.461255+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:29.461255+0000","last_undegraded":"2024-03-27T06:14:29.461255+0000","last_fullsized":"2024-03-27T06:14:29.461255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.049066+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:07.049066+0000","last_peered":"2024-03-27T06:14:07.049066+0000","last_clean":"2024-03-27T06:14:07.049066+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:07.049066+0000","last_undegraded":"2024-03-27T06:14:07.049066+0000","last_fullsized":"2024-03-27T06:14:07.049066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608165+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:18.608165+0000","last_peered":"2024-03-27T06:14:18.608165+0000","last_clean":"2024-03-27T06:14:18.608165+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:18.608165+0000","last_undegraded":"2024-03-27T06:14:18.608165+0000","last_fullsized":"2024-03-27T06:14:18.608165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808902+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:07.808902+0000","last_peered":"2024-03-27T06:14:07.808902+0000","last_clean":"2024-03-27T06:14:07.808902+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:07.808902+0000","last_undegraded":"2024-03-27T06:14:07.808902+0000","last_fullsized":"2024-03-27T06:14:07.808902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581161+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:13.581161+0000","last_peered":"2024-03-27T06:14:13.581161+0000","last_clean":"2024-03-27T06:14:13.581161+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:13.581161+0000","last_undegraded":"2024-03-27T06:14:13.581161+0000","last_fullsized":"2024-03-27T06:14:13.581161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613984+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:18.613984+0000","last_peered":"2024-03-27T06:14:18.613984+0000","last_clean":"2024-03-27T06:14:18.613984+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:18.613984+0000","last_undegraded":"2024-03-27T06:14:18.613984+0000","last_fullsized":"2024-03-27T06:14:18.613984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802187+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:06.802187+0000","last_peered":"2024-03-27T06:14:06.802187+0000","last_clean":"2024-03-27T06:14:06.802187+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:06.802187+0000","last_undegraded":"2024-03-27T06:14:06.802187+0000","last_fullsized":"2024-03-27T06:14:06.802187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.606607+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:18.606607+0000","last_peered":"2024-03-27T06:14:18.606607+0000","last_clean":"2024-03-27T06:14:18.606607+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:18.606607+0000","last_undegraded":"2024-03-27T06:14:18.606607+0000","last_fullsized":"2024-03-27T06:14:18.606607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.524195+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:08.524195+0000","last_peered":"2024-03-27T06:14:08.524195+0000","last_clean":"2024-03-27T06:14:08.524195+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:08.524195+0000","last_undegraded":"2024-03-27T06:14:08.524195+0000","last_fullsized":"2024-03-27T06:14:08.524195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.463061+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:07.463061+0000","last_peered":"2024-03-27T06:14:07.463061+0000","last_clean":"2024-03-27T06:14:07.463061+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:07.463061+0000","last_undegraded":"2024-03-27T06:14:07.463061+0000","last_fullsized":"2024-03-27T06:14:07.463061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.517650+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:08.517650+0000","last_peered":"2024-03-27T06:14:08.517650+0000","last_clean":"2024-03-27T06:14:08.517650+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:08.517650+0000","last_undegraded":"2024-03-27T06:14:08.517650+0000","last_fullsized":"2024-03-27T06:14:08.517650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047986+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:07.047986+0000","last_peered":"2024-03-27T06:14:07.047986+0000","last_clean":"2024-03-27T06:14:07.047986+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:07.047986+0000","last_undegraded":"2024-03-27T06:14:07.047986+0000","last_fullsized":"2024-03-27T06:14:07.047986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708579+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:06.708579+0000","last_peered":"2024-03-27T06:14:06.708579+0000","last_clean":"2024-03-27T06:14:06.708579+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:06.708579+0000","last_undegraded":"2024-03-27T06:14:06.708579+0000","last_fullsized":"2024-03-27T06:14:06.708579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048156+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:07.048156+0000","last_peered":"2024-03-27T06:14:07.048156+0000","last_clean":"2024-03-27T06:14:07.048156+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:07.048156+0000","last_undegraded":"2024-03-27T06:14:07.048156+0000","last_fullsized":"2024-03-27T06:14:07.048156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608234+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:18.608234+0000","last_peered":"2024-03-27T06:14:18.608234+0000","last_clean":"2024-03-27T06:14:18.608234+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:18.608234+0000","last_undegraded":"2024-03-27T06:14:18.608234+0000","last_fullsized":"2024-03-27T06:14:18.608234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801767+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:06.801767+0000","last_peered":"2024-03-27T06:14:06.801767+0000","last_clean":"2024-03-27T06:14:06.801767+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:06.801767+0000","last_undegraded":"2024-03-27T06:14:06.801767+0000","last_fullsized":"2024-03-27T06:14:06.801767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584731+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:13.584731+0000","last_peered":"2024-03-27T06:14:13.584731+0000","last_clean":"2024-03-27T06:14:13.584731+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:13.584731+0000","last_undegraded":"2024-03-27T06:14:13.584731+0000","last_fullsized":"2024-03-27T06:14:13.584731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608966+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:18.608966+0000","last_peered":"2024-03-27T06:14:18.608966+0000","last_clean":"2024-03-27T06:14:18.608966+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:18.608966+0000","last_undegraded":"2024-03-27T06:14:18.608966+0000","last_fullsized":"2024-03-27T06:14:18.608966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808793+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:07.808793+0000","last_peered":"2024-03-27T06:14:07.808793+0000","last_clean":"2024-03-27T06:14:07.808793+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:07.808793+0000","last_undegraded":"2024-03-27T06:14:07.808793+0000","last_fullsized":"2024-03-27T06:14:07.808793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.605510+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:18.605510+0000","last_peered":"2024-03-27T06:14:18.605510+0000","last_clean":"2024-03-27T06:14:18.605510+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:18.605510+0000","last_undegraded":"2024-03-27T06:14:18.605510+0000","last_fullsized":"2024-03-27T06:14:18.605510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708423+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:06.708423+0000","last_peered":"2024-03-27T06:14:06.708423+0000","last_clean":"2024-03-27T06:14:06.708423+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:06.708423+0000","last_undegraded":"2024-03-27T06:14:06.708423+0000","last_fullsized":"2024-03-27T06:14:06.708423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431429+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:05.431429+0000","last_peered":"2024-03-27T06:14:05.431429+0000","last_clean":"2024-03-27T06:14:05.431429+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:05.431429+0000","last_undegraded":"2024-03-27T06:14:05.431429+0000","last_fullsized":"2024-03-27T06:14:05.431429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048444+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:07.048444+0000","last_peered":"2024-03-27T06:14:07.048444+0000","last_clean":"2024-03-27T06:14:07.048444+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:07.048444+0000","last_undegraded":"2024-03-27T06:14:07.048444+0000","last_fullsized":"2024-03-27T06:14:07.048444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708415+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:06.708415+0000","last_peered":"2024-03-27T06:14:06.708415+0000","last_clean":"2024-03-27T06:14:06.708415+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:06.708415+0000","last_undegraded":"2024-03-27T06:14:06.708415+0000","last_fullsized":"2024-03-27T06:14:06.708415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462059+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:07.462059+0000","last_peered":"2024-03-27T06:14:07.462059+0000","last_clean":"2024-03-27T06:14:07.462059+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:07.462059+0000","last_undegraded":"2024-03-27T06:14:07.462059+0000","last_fullsized":"2024-03-27T06:14:07.462059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431591+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:05.431591+0000","last_peered":"2024-03-27T06:14:05.431591+0000","last_clean":"2024-03-27T06:14:05.431591+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:05.431591+0000","last_undegraded":"2024-03-27T06:14:05.431591+0000","last_fullsized":"2024-03-27T06:14:05.431591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809116+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:07.809116+0000","last_peered":"2024-03-27T06:14:07.809116+0000","last_clean":"2024-03-27T06:14:07.809116+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:07.809116+0000","last_undegraded":"2024-03-27T06:14:07.809116+0000","last_fullsized":"2024-03-27T06:14:07.809116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613716+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:18.613716+0000","last_peered":"2024-03-27T06:14:18.613716+0000","last_clean":"2024-03-27T06:14:18.613716+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:18.613716+0000","last_undegraded":"2024-03-27T06:14:18.613716+0000","last_fullsized":"2024-03-27T06:14:18.613716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046680+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:07.046680+0000","last_peered":"2024-03-27T06:14:07.046680+0000","last_clean":"2024-03-27T06:14:07.046680+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:07.046680+0000","last_undegraded":"2024-03-27T06:14:07.046680+0000","last_fullsized":"2024-03-27T06:14:07.046680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581236+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:13.581236+0000","last_peered":"2024-03-27T06:14:13.581236+0000","last_clean":"2024-03-27T06:14:13.581236+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:13.581236+0000","last_undegraded":"2024-03-27T06:14:13.581236+0000","last_fullsized":"2024-03-27T06:14:13.581236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046583+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:07.046583+0000","last_peered":"2024-03-27T06:14:07.046583+0000","last_clean":"2024-03-27T06:14:07.046583+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:07.046583+0000","last_undegraded":"2024-03-27T06:14:07.046583+0000","last_fullsized":"2024-03-27T06:14:07.046583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584347+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:13.584347+0000","last_peered":"2024-03-27T06:14:13.584347+0000","last_clean":"2024-03-27T06:14:13.584347+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:13.584347+0000","last_undegraded":"2024-03-27T06:14:13.584347+0000","last_fullsized":"2024-03-27T06:14:13.584347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582654+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:13.582654+0000","last_peered":"2024-03-27T06:14:13.582654+0000","last_clean":"2024-03-27T06:14:13.582654+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:13.582654+0000","last_undegraded":"2024-03-27T06:14:13.582654+0000","last_fullsized":"2024-03-27T06:14:13.582654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809031+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:07.809031+0000","last_peered":"2024-03-27T06:14:07.809031+0000","last_clean":"2024-03-27T06:14:07.809031+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:07.809031+0000","last_undegraded":"2024-03-27T06:14:07.809031+0000","last_fullsized":"2024-03-27T06:14:07.809031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.580898+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:13.580898+0000","last_peered":"2024-03-27T06:14:13.580898+0000","last_clean":"2024-03-27T06:14:13.580898+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:13.580898+0000","last_undegraded":"2024-03-27T06:14:13.580898+0000","last_fullsized":"2024-03-27T06:14:13.580898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525809+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:08.525809+0000","last_peered":"2024-03-27T06:14:08.525809+0000","last_clean":"2024-03-27T06:14:08.525809+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:08.525809+0000","last_undegraded":"2024-03-27T06:14:08.525809+0000","last_fullsized":"2024-03-27T06:14:08.525809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802159+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:06.802159+0000","last_peered":"2024-03-27T06:14:06.802159+0000","last_clean":"2024-03-27T06:14:06.802159+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:06.802159+0000","last_undegraded":"2024-03-27T06:14:06.802159+0000","last_fullsized":"2024-03-27T06:14:06.802159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613242+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:18.613242+0000","last_peered":"2024-03-27T06:14:18.613242+0000","last_clean":"2024-03-27T06:14:18.613242+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:18.613242+0000","last_undegraded":"2024-03-27T06:14:18.613242+0000","last_fullsized":"2024-03-27T06:14:18.613242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708697+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:06.708697+0000","last_peered":"2024-03-27T06:14:06.708697+0000","last_clean":"2024-03-27T06:14:06.708697+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:06.708697+0000","last_undegraded":"2024-03-27T06:14:06.708697+0000","last_fullsized":"2024-03-27T06:14:06.708697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.515193+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:08.515193+0000","last_peered":"2024-03-27T06:14:08.515193+0000","last_clean":"2024-03-27T06:14:08.515193+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:08.515193+0000","last_undegraded":"2024-03-27T06:14:08.515193+0000","last_fullsized":"2024-03-27T06:14:08.515193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708688+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:06.708688+0000","last_peered":"2024-03-27T06:14:06.708688+0000","last_clean":"2024-03-27T06:14:06.708688+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:06.708688+0000","last_undegraded":"2024-03-27T06:14:06.708688+0000","last_fullsized":"2024-03-27T06:14:06.708688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613729+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:18.613729+0000","last_peered":"2024-03-27T06:14:18.613729+0000","last_clean":"2024-03-27T06:14:18.613729+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:18.613729+0000","last_undegraded":"2024-03-27T06:14:18.613729+0000","last_fullsized":"2024-03-27T06:14:18.613729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5476,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":10952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":666777,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":79039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666133,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:35.627 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-03-27T06:14:35.627 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:14:35.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.872+0000 7f3dae000640 1 Processor -- start 2024-03-27T06:14:35.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.872+0000 7f3dae000640 1 -- start start 2024-03-27T06:14:35.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.874+0000 7f3dae000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3da80d0970 0x7f3da815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3dae000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3da80cb440 0x7f3da80c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3dae000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3da80cb9f0 con 0x7f3da80d0d50 2024-03-27T06:14:35.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3dae000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f3da815e2e0 con 0x7f3da80d0970 2024-03-27T06:14:35.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3dae000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3da815e5c0 con 0x7f3da80cb440 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3da80cb440 0x7f3da80c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da6a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3da80d0d50 0x7f3da8154aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39926/0 (socket says 172.21.15.132:39926) 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da6a00640 1 -- 172.21.15.132:0/1752810807 learned_addr learned my addr 172.21.15.132:0/1752810807 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7400640 1 --2- 172.21.15.132:0/1752810807 >> v2:172.21.15.132:3300/0 conn(0x7f3da80d0970 0x7f3da815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da6000640 1 -- 172.21.15.132:0/1752810807 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131950174 0 0) 0x7f3da80cb9f0 con 0x7f3da80d0d50 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da6000640 1 -- 172.21.15.132:0/1752810807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c003520 con 0x7f3da80d0d50 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7e00640 1 -- 172.21.15.132:0/1752810807 >> v2:172.21.15.132:3300/0 conn(0x7f3da80d0970 msgr2=0x7f3da815b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7e00640 1 --2- 172.21.15.132:0/1752810807 >> v2:172.21.15.132:3300/0 conn(0x7f3da80d0970 0x7f3da815b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7e00640 1 -- 172.21.15.132:0/1752810807 >> v1:172.21.15.134:6789/0 conn(0x7f3da80d0d50 legacy=0x7f3da8154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.875+0000 7f3da7e00640 1 -- 172.21.15.132:0/1752810807 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da815e8a0 con 0x7f3da80cb440 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.877+0000 7f3da7e00640 1 --2- 172.21.15.132:0/1752810807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80cb440 0x7f3da80c8a40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3da815ee60 tx=0x7f3da002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=449f5c7d6266001e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.877+0000 7f3da6000640 1 -- 172.21.15.132:0/1752810807 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0039070 con 0x7f3da80cb440 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.877+0000 7f3da6000640 1 -- 172.21.15.132:0/1752810807 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da002cc80 con 0x7f3da80cb440 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.877+0000 7f3da6000640 1 -- 172.21.15.132:0/1752810807 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0034660 con 0x7f3da80cb440 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 -- 172.21.15.132:0/1752810807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80cb440 msgr2=0x7f3da80c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 --2- 172.21.15.132:0/1752810807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80cb440 0x7f3da80c8a40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3da815ee60 tx=0x7f3da002bed0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 -- 172.21.15.132:0/1752810807 shutdown_connections 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 --2- 172.21.15.132:0/1752810807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80cb440 0x7f3da80c8a40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 --2- 172.21.15.132:0/1752810807 >> v2:172.21.15.132:3300/0 conn(0x7f3da80d0970 0x7f3da815b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 -- 172.21.15.132:0/1752810807 >> 172.21.15.132:0/1752810807 conn(0x7f3da806dbd0 msgr2=0x7f3da806dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 -- 172.21.15.132:0/1752810807 shutdown_connections 2024-03-27T06:14:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.878+0000 7f3dae000640 1 -- 172.21.15.132:0/1752810807 wait complete. 2024-03-27T06:14:35.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.879+0000 7f3dae000640 1 Processor -- start 2024-03-27T06:14:35.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.880+0000 7f3dae000640 1 -- start start 2024-03-27T06:14:35.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.880+0000 7f3dae000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 0x7f3da815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.880+0000 7f3dae000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 0x7f3da8151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.880+0000 7f3dae000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80d0d50 0x7f3da814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3dae000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3da8155330 con 0x7f3da80cb440 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3dae000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3da81551b0 con 0x7f3da80d0d50 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3dae000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3da80cc7f0 con 0x7f3da80d0970 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 0x7f3da815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 0x7f3da8151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80d0d50 0x7f3da814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 0x7f3da815ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52892/0 (socket says 172.21.15.132:52892) 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da7400640 1 -- 172.21.15.132:0/2274100484 learned_addr learned my addr 172.21.15.132:0/2274100484 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 -- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80d0d50 msgr2=0x7f3da814e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80d0d50 0x7f3da814e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 -- 172.21.15.132:0/2274100484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 msgr2=0x7f3da815ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:35.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 0x7f3da815ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 -- 172.21.15.132:0/2274100484 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da815dab0 con 0x7f3da80d0970 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.881+0000 7f3da6a00640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 0x7f3da8151a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c390 tx=0x7f3d94001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.882+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94017070 con 0x7f3da80d0970 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.882+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d94005ce0 con 0x7f3da80d0970 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.882+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940072d0 con 0x7f3da80d0970 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.882+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da815dd90 con 0x7f3da80d0970 2024-03-27T06:14:35.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.882+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da80cfeb0 con 0x7f3da80d0970 2024-03-27T06:14:35.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.883+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da815bc40 con 0x7f3da80d0970 2024-03-27T06:14:35.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.883+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3d9401a070 con 0x7f3da80d0970 2024-03-27T06:14:35.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.884+0000 7f3d9f400640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 0x7f3d8806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:35.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.884+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3d9408a900 con 0x7f3da80d0970 2024-03-27T06:14:35.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.885+0000 7f3da7400640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 0x7f3d8806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:35.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.885+0000 7f3da7400640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 0x7f3d8806e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009c00 tx=0x7f3d98009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:35.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:35.889+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3d940588c0 con 0x7f3da80d0970 2024-03-27T06:14:36.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.075+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3da80ce7c0 con 0x7f3d8806c530 2024-03-27T06:14:36.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.082+0000 7f3d9f400640 1 -- 172.21.15.132:0/2274100484 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251989 (secure 0 0 0) 0x7f3da80ce7c0 con 0x7f3d8806c530 2024-03-27T06:14:36.077 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:36.078 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 msgr2=0x7f3d8806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 0x7f3d8806e9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009c00 tx=0x7f3d98009190 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 msgr2=0x7f3da8151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 0x7f3da8151a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c390 tx=0x7f3d94001e70 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 shutdown_connections 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3d8806c530 0x7f3d8806e9f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3da80d0d50 0x7f3da814e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3da80d0970 0x7f3da8151a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 --2- 172.21.15.132:0/2274100484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80cb440 0x7f3da815ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 >> 172.21.15.132:0/2274100484 conn(0x7f3da806dbd0 msgr2=0x7f3da80ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 shutdown_connections 2024-03-27T06:14:36.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.087+0000 7f3dae000640 1 -- 172.21.15.132:0/2274100484 wait complete. 2024-03-27T06:14:36.096 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-27T06:14:34.397027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595844,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":286,"num_write_kb":2456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218112,"kb_used_data":1984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756608,"statfs":{"total":773094113280,"available":772870766592,"internally_reserved":0,"allocated":2031616,"data_stored":1796528,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048058+0000","last_change":"2024-03-27T06:14:03.555851+0000","last_active":"2024-03-27T06:14:07.048058+0000","last_peered":"2024-03-27T06:14:07.048058+0000","last_clean":"2024-03-27T06:14:07.048058+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:14:07.048058+0000","last_undegraded":"2024-03-27T06:14:07.048058+0000","last_fullsized":"2024-03-27T06:14:07.048058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708723+0000","last_change":"2024-03-27T06:14:03.524284+0000","last_active":"2024-03-27T06:14:06.708723+0000","last_peered":"2024-03-27T06:14:06.708723+0000","last_clean":"2024-03-27T06:14:06.708723+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:14:06.708723+0000","last_undegraded":"2024-03-27T06:14:06.708723+0000","last_fullsized":"2024-03-27T06:14:06.708723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708489+0000","last_change":"2024-03-27T06:14:03.528945+0000","last_active":"2024-03-27T06:14:06.708489+0000","last_peered":"2024-03-27T06:14:06.708489+0000","last_clean":"2024-03-27T06:14:06.708489+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:14:06.708489+0000","last_undegraded":"2024-03-27T06:14:06.708489+0000","last_fullsized":"2024-03-27T06:14:06.708489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462134+0000","last_change":"2024-03-27T06:14:03.530013+0000","last_active":"2024-03-27T06:14:07.462134+0000","last_peered":"2024-03-27T06:14:07.462134+0000","last_clean":"2024-03-27T06:14:07.462134+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:14:07.462134+0000","last_undegraded":"2024-03-27T06:14:07.462134+0000","last_fullsized":"2024-03-27T06:14:07.462134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048224+0000","last_change":"2024-03-27T06:14:03.547365+0000","last_active":"2024-03-27T06:14:07.048224+0000","last_peered":"2024-03-27T06:14:07.048224+0000","last_clean":"2024-03-27T06:14:07.048224+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:14:07.048224+0000","last_undegraded":"2024-03-27T06:14:07.048224+0000","last_fullsized":"2024-03-27T06:14:07.048224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462236+0000","last_change":"2024-03-27T06:14:03.531530+0000","last_active":"2024-03-27T06:14:07.462236+0000","last_peered":"2024-03-27T06:14:07.462236+0000","last_clean":"2024-03-27T06:14:07.462236+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:14:07.462236+0000","last_undegraded":"2024-03-27T06:14:07.462236+0000","last_fullsized":"2024-03-27T06:14:07.462236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047358+0000","last_change":"2024-03-27T06:14:03.546346+0000","last_active":"2024-03-27T06:14:07.047358+0000","last_peered":"2024-03-27T06:14:07.047358+0000","last_clean":"2024-03-27T06:14:07.047358+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:14:07.047358+0000","last_undegraded":"2024-03-27T06:14:07.047358+0000","last_fullsized":"2024-03-27T06:14:07.047358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708492+0000","last_change":"2024-03-27T06:14:03.542154+0000","last_active":"2024-03-27T06:14:06.708492+0000","last_peered":"2024-03-27T06:14:06.708492+0000","last_clean":"2024-03-27T06:14:06.708492+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:14:06.708492+0000","last_undegraded":"2024-03-27T06:14:06.708492+0000","last_fullsized":"2024-03-27T06:14:06.708492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802323+0000","last_change":"2024-03-27T06:14:03.540805+0000","last_active":"2024-03-27T06:14:06.802323+0000","last_peered":"2024-03-27T06:14:06.802323+0000","last_clean":"2024-03-27T06:14:06.802323+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:14:06.802323+0000","last_undegraded":"2024-03-27T06:14:06.802323+0000","last_fullsized":"2024-03-27T06:14:06.802323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048760+0000","last_change":"2024-03-27T06:14:03.553940+0000","last_active":"2024-03-27T06:14:07.048760+0000","last_peered":"2024-03-27T06:14:07.048760+0000","last_clean":"2024-03-27T06:14:07.048760+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:14:07.048760+0000","last_undegraded":"2024-03-27T06:14:07.048760+0000","last_fullsized":"2024-03-27T06:14:07.048760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048755+0000","last_change":"2024-03-27T06:14:03.551695+0000","last_active":"2024-03-27T06:14:07.048755+0000","last_peered":"2024-03-27T06:14:07.048755+0000","last_clean":"2024-03-27T06:14:07.048755+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:14:07.048755+0000","last_undegraded":"2024-03-27T06:14:07.048755+0000","last_fullsized":"2024-03-27T06:14:07.048755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047765+0000","last_change":"2024-03-27T06:14:03.553218+0000","last_active":"2024-03-27T06:14:07.047765+0000","last_peered":"2024-03-27T06:14:07.047765+0000","last_clean":"2024-03-27T06:14:07.047765+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:14:07.047765+0000","last_undegraded":"2024-03-27T06:14:07.047765+0000","last_fullsized":"2024-03-27T06:14:07.047765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808956+0000","last_change":"2024-03-27T06:14:03.552612+0000","last_active":"2024-03-27T06:14:07.808956+0000","last_peered":"2024-03-27T06:14:07.808956+0000","last_clean":"2024-03-27T06:14:07.808956+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:14:07.808956+0000","last_undegraded":"2024-03-27T06:14:07.808956+0000","last_fullsized":"2024-03-27T06:14:07.808956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808841+0000","last_change":"2024-03-27T06:14:03.516724+0000","last_active":"2024-03-27T06:14:07.808841+0000","last_peered":"2024-03-27T06:14:07.808841+0000","last_clean":"2024-03-27T06:14:07.808841+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:14:07.808841+0000","last_undegraded":"2024-03-27T06:14:07.808841+0000","last_fullsized":"2024-03-27T06:14:07.808841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708767+0000","last_change":"2024-03-27T06:14:03.547481+0000","last_active":"2024-03-27T06:14:06.708767+0000","last_peered":"2024-03-27T06:14:06.708767+0000","last_clean":"2024-03-27T06:14:06.708767+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:14:06.708767+0000","last_undegraded":"2024-03-27T06:14:06.708767+0000","last_fullsized":"2024-03-27T06:14:06.708767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529610+0000","last_change":"2024-03-27T06:14:03.539582+0000","last_active":"2024-03-27T06:14:04.529610+0000","last_peered":"2024-03-27T06:14:04.529610+0000","last_clean":"2024-03-27T06:14:04.529610+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:14:04.529610+0000","last_undegraded":"2024-03-27T06:14:04.529610+0000","last_fullsized":"2024-03-27T06:14:04.529610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531077+0000","last_change":"2024-03-27T06:14:03.558058+0000","last_active":"2024-03-27T06:14:04.531077+0000","last_peered":"2024-03-27T06:14:04.531077+0000","last_clean":"2024-03-27T06:14:04.531077+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:14:04.531077+0000","last_undegraded":"2024-03-27T06:14:04.531077+0000","last_fullsized":"2024-03-27T06:14:04.531077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:38:42.303978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527579+0000","last_change":"2024-03-27T06:14:03.530857+0000","last_active":"2024-03-27T06:14:04.527579+0000","last_peered":"2024-03-27T06:14:04.527579+0000","last_clean":"2024-03-27T06:14:04.527579+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:14:04.527579+0000","last_undegraded":"2024-03-27T06:14:04.527579+0000","last_fullsized":"2024-03-27T06:14:04.527579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530396+0000","last_change":"2024-03-27T06:14:03.540277+0000","last_active":"2024-03-27T06:14:04.530396+0000","last_peered":"2024-03-27T06:14:04.530396+0000","last_clean":"2024-03-27T06:14:04.530396+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:14:04.530396+0000","last_undegraded":"2024-03-27T06:14:04.530396+0000","last_fullsized":"2024-03-27T06:14:04.530396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:52.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048108+0000","last_change":"2024-03-27T06:14:03.550870+0000","last_active":"2024-03-27T06:14:07.048108+0000","last_peered":"2024-03-27T06:14:07.048108+0000","last_clean":"2024-03-27T06:14:07.048108+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:14:07.048108+0000","last_undegraded":"2024-03-27T06:14:07.048108+0000","last_fullsized":"2024-03-27T06:14:07.048108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:25:34.734765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046295+0000","last_change":"2024-03-27T06:14:03.528619+0000","last_active":"2024-03-27T06:14:07.046295+0000","last_peered":"2024-03-27T06:14:07.046295+0000","last_clean":"2024-03-27T06:14:07.046295+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:14:07.046295+0000","last_undegraded":"2024-03-27T06:14:07.046295+0000","last_fullsized":"2024-03-27T06:14:07.046295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:59:34.952622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801932+0000","last_change":"2024-03-27T06:14:03.558379+0000","last_active":"2024-03-27T06:14:06.801932+0000","last_peered":"2024-03-27T06:14:06.801932+0000","last_clean":"2024-03-27T06:14:06.801932+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:14:06.801932+0000","last_undegraded":"2024-03-27T06:14:06.801932+0000","last_fullsized":"2024-03-27T06:14:06.801932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708558+0000","last_change":"2024-03-27T06:14:03.530142+0000","last_active":"2024-03-27T06:14:06.708558+0000","last_peered":"2024-03-27T06:14:06.708558+0000","last_clean":"2024-03-27T06:14:06.708558+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:14:06.708558+0000","last_undegraded":"2024-03-27T06:14:06.708558+0000","last_fullsized":"2024-03-27T06:14:06.708558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529584+0000","last_change":"2024-03-27T06:14:03.545291+0000","last_active":"2024-03-27T06:14:04.529584+0000","last_peered":"2024-03-27T06:14:04.529584+0000","last_clean":"2024-03-27T06:14:04.529584+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:14:04.529584+0000","last_undegraded":"2024-03-27T06:14:04.529584+0000","last_fullsized":"2024-03-27T06:14:04.529584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708736+0000","last_change":"2024-03-27T06:14:03.537705+0000","last_active":"2024-03-27T06:14:06.708736+0000","last_peered":"2024-03-27T06:14:06.708736+0000","last_clean":"2024-03-27T06:14:06.708736+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:14:06.708736+0000","last_undegraded":"2024-03-27T06:14:06.708736+0000","last_fullsized":"2024-03-27T06:14:06.708736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462807+0000","last_change":"2024-03-27T06:14:03.543752+0000","last_active":"2024-03-27T06:14:07.462807+0000","last_peered":"2024-03-27T06:14:07.462807+0000","last_clean":"2024-03-27T06:14:07.462807+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:14:07.462807+0000","last_undegraded":"2024-03-27T06:14:07.462807+0000","last_fullsized":"2024-03-27T06:14:07.462807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:56.712151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809252+0000","last_change":"2024-03-27T06:14:03.516309+0000","last_active":"2024-03-27T06:14:07.809252+0000","last_peered":"2024-03-27T06:14:07.809252+0000","last_clean":"2024-03-27T06:14:07.809252+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:14:07.809252+0000","last_undegraded":"2024-03-27T06:14:07.809252+0000","last_fullsized":"2024-03-27T06:14:07.809252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:29.119473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809134+0000","last_change":"2024-03-27T06:14:03.557454+0000","last_active":"2024-03-27T06:14:07.809134+0000","last_peered":"2024-03-27T06:14:07.809134+0000","last_clean":"2024-03-27T06:14:07.809134+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:14:07.809134+0000","last_undegraded":"2024-03-27T06:14:07.809134+0000","last_fullsized":"2024-03-27T06:14:07.809134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:55.029903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708534+0000","last_change":"2024-03-27T06:14:03.524688+0000","last_active":"2024-03-27T06:14:06.708534+0000","last_peered":"2024-03-27T06:14:06.708534+0000","last_clean":"2024-03-27T06:14:06.708534+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:14:06.708534+0000","last_undegraded":"2024-03-27T06:14:06.708534+0000","last_fullsized":"2024-03-27T06:14:06.708534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529938+0000","last_change":"2024-03-27T06:14:03.539468+0000","last_active":"2024-03-27T06:14:04.529938+0000","last_peered":"2024-03-27T06:14:04.529938+0000","last_clean":"2024-03-27T06:14:04.529938+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:14:04.529938+0000","last_undegraded":"2024-03-27T06:14:04.529938+0000","last_fullsized":"2024-03-27T06:14:04.529938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527407+0000","last_change":"2024-03-27T06:14:03.528094+0000","last_active":"2024-03-27T06:14:04.527407+0000","last_peered":"2024-03-27T06:14:04.527407+0000","last_clean":"2024-03-27T06:14:04.527407+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:14:04.527407+0000","last_undegraded":"2024-03-27T06:14:04.527407+0000","last_fullsized":"2024-03-27T06:14:04.527407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:34.176304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.461979+0000","last_change":"2024-03-27T06:14:03.527162+0000","last_active":"2024-03-27T06:14:07.461979+0000","last_peered":"2024-03-27T06:14:07.461979+0000","last_clean":"2024-03-27T06:14:07.461979+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:14:07.461979+0000","last_undegraded":"2024-03-27T06:14:07.461979+0000","last_fullsized":"2024-03-27T06:14:07.461979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802104+0000","last_change":"2024-03-27T06:14:03.534716+0000","last_active":"2024-03-27T06:14:06.802104+0000","last_peered":"2024-03-27T06:14:06.802104+0000","last_clean":"2024-03-27T06:14:06.802104+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:14:06.802104+0000","last_undegraded":"2024-03-27T06:14:06.802104+0000","last_fullsized":"2024-03-27T06:14:06.802104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:48.583490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.577932+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:14:13.577932+0000","last_peered":"2024-03-27T06:14:13.577932+0000","last_clean":"2024-03-27T06:14:13.577932+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:14:13.577932+0000","last_undegraded":"2024-03-27T06:14:13.577932+0000","last_fullsized":"2024-03-27T06:14:13.577932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046346+0000","last_change":"2024-03-27T06:14:03.547877+0000","last_active":"2024-03-27T06:14:07.046346+0000","last_peered":"2024-03-27T06:14:07.046346+0000","last_clean":"2024-03-27T06:14:07.046346+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:14:07.046346+0000","last_undegraded":"2024-03-27T06:14:07.046346+0000","last_fullsized":"2024-03-27T06:14:07.046346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:51.778368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608838+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:14:18.608838+0000","last_peered":"2024-03-27T06:14:18.608838+0000","last_clean":"2024-03-27T06:14:18.608838+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:14:18.608838+0000","last_undegraded":"2024-03-27T06:14:18.608838+0000","last_fullsized":"2024-03-27T06:14:18.608838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608211+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:14:18.608211+0000","last_peered":"2024-03-27T06:14:18.608211+0000","last_clean":"2024-03-27T06:14:18.608211+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:14:18.608211+0000","last_undegraded":"2024-03-27T06:14:18.608211+0000","last_fullsized":"2024-03-27T06:14:18.608211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708759+0000","last_change":"2024-03-27T06:14:03.525257+0000","last_active":"2024-03-27T06:14:06.708759+0000","last_peered":"2024-03-27T06:14:06.708759+0000","last_clean":"2024-03-27T06:14:06.708759+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:14:06.708759+0000","last_undegraded":"2024-03-27T06:14:06.708759+0000","last_fullsized":"2024-03-27T06:14:06.708759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431193+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:14:05.431193+0000","last_peered":"2024-03-27T06:14:05.431193+0000","last_clean":"2024-03-27T06:14:05.431193+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:14:05.431193+0000","last_undegraded":"2024-03-27T06:14:05.431193+0000","last_fullsized":"2024-03-27T06:14:05.431193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046377+0000","last_change":"2024-03-27T06:14:03.530899+0000","last_active":"2024-03-27T06:14:07.046377+0000","last_peered":"2024-03-27T06:14:07.046377+0000","last_clean":"2024-03-27T06:14:07.046377+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:14:07.046377+0000","last_undegraded":"2024-03-27T06:14:07.046377+0000","last_fullsized":"2024-03-27T06:14:07.046377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:01:59.091825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.579962+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:14:13.579962+0000","last_peered":"2024-03-27T06:14:13.579962+0000","last_clean":"2024-03-27T06:14:13.579962+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:14:13.579962+0000","last_undegraded":"2024-03-27T06:14:13.579962+0000","last_fullsized":"2024-03-27T06:14:13.579962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462989+0000","last_change":"2024-03-27T06:14:03.543946+0000","last_active":"2024-03-27T06:14:07.462989+0000","last_peered":"2024-03-27T06:14:07.462989+0000","last_clean":"2024-03-27T06:14:07.462989+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:14:07.462989+0000","last_undegraded":"2024-03-27T06:14:07.462989+0000","last_fullsized":"2024-03-27T06:14:07.462989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582879+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:14:13.582879+0000","last_peered":"2024-03-27T06:14:13.582879+0000","last_clean":"2024-03-27T06:14:13.582879+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:14:13.582879+0000","last_undegraded":"2024-03-27T06:14:13.582879+0000","last_fullsized":"2024-03-27T06:14:13.582879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708885+0000","last_change":"2024-03-27T06:14:03.542328+0000","last_active":"2024-03-27T06:14:06.708885+0000","last_peered":"2024-03-27T06:14:06.708885+0000","last_clean":"2024-03-27T06:14:06.708885+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:14:06.708885+0000","last_undegraded":"2024-03-27T06:14:06.708885+0000","last_fullsized":"2024-03-27T06:14:06.708885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:40.972841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.516309+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:14:08.516309+0000","last_peered":"2024-03-27T06:14:08.516309+0000","last_clean":"2024-03-27T06:14:08.516309+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:14:08.516309+0000","last_undegraded":"2024-03-27T06:14:08.516309+0000","last_fullsized":"2024-03-27T06:14:08.516309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529990+0000","last_change":"2024-03-27T06:14:03.546687+0000","last_active":"2024-03-27T06:14:04.529990+0000","last_peered":"2024-03-27T06:14:04.529990+0000","last_clean":"2024-03-27T06:14:04.529990+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:14:04.529990+0000","last_undegraded":"2024-03-27T06:14:04.529990+0000","last_fullsized":"2024-03-27T06:14:04.529990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:50:31.856111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:29.461255+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:14:29.461255+0000","last_peered":"2024-03-27T06:14:29.461255+0000","last_clean":"2024-03-27T06:14:29.461255+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:14:29.461255+0000","last_undegraded":"2024-03-27T06:14:29.461255+0000","last_fullsized":"2024-03-27T06:14:29.461255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.049066+0000","last_change":"2024-03-27T06:14:03.551754+0000","last_active":"2024-03-27T06:14:07.049066+0000","last_peered":"2024-03-27T06:14:07.049066+0000","last_clean":"2024-03-27T06:14:07.049066+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:14:07.049066+0000","last_undegraded":"2024-03-27T06:14:07.049066+0000","last_fullsized":"2024-03-27T06:14:07.049066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:34.458248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608165+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:14:18.608165+0000","last_peered":"2024-03-27T06:14:18.608165+0000","last_clean":"2024-03-27T06:14:18.608165+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:14:18.608165+0000","last_undegraded":"2024-03-27T06:14:18.608165+0000","last_fullsized":"2024-03-27T06:14:18.608165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808902+0000","last_change":"2024-03-27T06:14:03.538233+0000","last_active":"2024-03-27T06:14:07.808902+0000","last_peered":"2024-03-27T06:14:07.808902+0000","last_clean":"2024-03-27T06:14:07.808902+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:14:07.808902+0000","last_undegraded":"2024-03-27T06:14:07.808902+0000","last_fullsized":"2024-03-27T06:14:07.808902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:29.951895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581161+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:14:13.581161+0000","last_peered":"2024-03-27T06:14:13.581161+0000","last_clean":"2024-03-27T06:14:13.581161+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:14:13.581161+0000","last_undegraded":"2024-03-27T06:14:13.581161+0000","last_fullsized":"2024-03-27T06:14:13.581161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.531386+0000","last_change":"2024-03-27T06:14:03.555955+0000","last_active":"2024-03-27T06:14:04.531386+0000","last_peered":"2024-03-27T06:14:04.531386+0000","last_clean":"2024-03-27T06:14:04.531386+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:14:04.531386+0000","last_undegraded":"2024-03-27T06:14:04.531386+0000","last_fullsized":"2024-03-27T06:14:04.531386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:46:31.124472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613984+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:14:18.613984+0000","last_peered":"2024-03-27T06:14:18.613984+0000","last_clean":"2024-03-27T06:14:18.613984+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:14:18.613984+0000","last_undegraded":"2024-03-27T06:14:18.613984+0000","last_fullsized":"2024-03-27T06:14:18.613984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802187+0000","last_change":"2024-03-27T06:14:03.536335+0000","last_active":"2024-03-27T06:14:06.802187+0000","last_peered":"2024-03-27T06:14:06.802187+0000","last_clean":"2024-03-27T06:14:06.802187+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:14:06.802187+0000","last_undegraded":"2024-03-27T06:14:06.802187+0000","last_fullsized":"2024-03-27T06:14:06.802187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:50.912723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.606607+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:14:18.606607+0000","last_peered":"2024-03-27T06:14:18.606607+0000","last_clean":"2024-03-27T06:14:18.606607+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:14:18.606607+0000","last_undegraded":"2024-03-27T06:14:18.606607+0000","last_fullsized":"2024-03-27T06:14:18.606607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529668+0000","last_change":"2024-03-27T06:14:03.540296+0000","last_active":"2024-03-27T06:14:04.529668+0000","last_peered":"2024-03-27T06:14:04.529668+0000","last_clean":"2024-03-27T06:14:04.529668+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:14:04.529668+0000","last_undegraded":"2024-03-27T06:14:04.529668+0000","last_fullsized":"2024-03-27T06:14:04.529668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:19.478500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.524195+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:14:08.524195+0000","last_peered":"2024-03-27T06:14:08.524195+0000","last_clean":"2024-03-27T06:14:08.524195+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:14:08.524195+0000","last_undegraded":"2024-03-27T06:14:08.524195+0000","last_fullsized":"2024-03-27T06:14:08.524195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.463061+0000","last_change":"2024-03-27T06:14:03.552678+0000","last_active":"2024-03-27T06:14:07.463061+0000","last_peered":"2024-03-27T06:14:07.463061+0000","last_clean":"2024-03-27T06:14:07.463061+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:14:07.463061+0000","last_undegraded":"2024-03-27T06:14:07.463061+0000","last_fullsized":"2024-03-27T06:14:07.463061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:02:44.264648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.517650+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:14:08.517650+0000","last_peered":"2024-03-27T06:14:08.517650+0000","last_clean":"2024-03-27T06:14:08.517650+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:14:08.517650+0000","last_undegraded":"2024-03-27T06:14:08.517650+0000","last_fullsized":"2024-03-27T06:14:08.517650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.047986+0000","last_change":"2024-03-27T06:14:03.551987+0000","last_active":"2024-03-27T06:14:07.047986+0000","last_peered":"2024-03-27T06:14:07.047986+0000","last_clean":"2024-03-27T06:14:07.047986+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:14:07.047986+0000","last_undegraded":"2024-03-27T06:14:07.047986+0000","last_fullsized":"2024-03-27T06:14:07.047986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708579+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:14:06.708579+0000","last_peered":"2024-03-27T06:14:06.708579+0000","last_clean":"2024-03-27T06:14:06.708579+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:14:06.708579+0000","last_undegraded":"2024-03-27T06:14:06.708579+0000","last_fullsized":"2024-03-27T06:14:06.708579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048156+0000","last_change":"2024-03-27T06:14:03.550381+0000","last_active":"2024-03-27T06:14:07.048156+0000","last_peered":"2024-03-27T06:14:07.048156+0000","last_clean":"2024-03-27T06:14:07.048156+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:14:07.048156+0000","last_undegraded":"2024-03-27T06:14:07.048156+0000","last_fullsized":"2024-03-27T06:14:07.048156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:25:18.704106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608234+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:14:18.608234+0000","last_peered":"2024-03-27T06:14:18.608234+0000","last_clean":"2024-03-27T06:14:18.608234+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:14:18.608234+0000","last_undegraded":"2024-03-27T06:14:18.608234+0000","last_fullsized":"2024-03-27T06:14:18.608234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530218+0000","last_change":"2024-03-27T06:14:03.543968+0000","last_active":"2024-03-27T06:14:04.530218+0000","last_peered":"2024-03-27T06:14:04.530218+0000","last_clean":"2024-03-27T06:14:04.530218+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:14:04.530218+0000","last_undegraded":"2024-03-27T06:14:04.530218+0000","last_fullsized":"2024-03-27T06:14:04.530218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:10.616601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.801767+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:14:06.801767+0000","last_peered":"2024-03-27T06:14:06.801767+0000","last_clean":"2024-03-27T06:14:06.801767+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:14:06.801767+0000","last_undegraded":"2024-03-27T06:14:06.801767+0000","last_fullsized":"2024-03-27T06:14:06.801767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584731+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:14:13.584731+0000","last_peered":"2024-03-27T06:14:13.584731+0000","last_clean":"2024-03-27T06:14:13.584731+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:14:13.584731+0000","last_undegraded":"2024-03-27T06:14:13.584731+0000","last_fullsized":"2024-03-27T06:14:13.584731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.541371+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.608966+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:14:18.608966+0000","last_peered":"2024-03-27T06:14:18.608966+0000","last_clean":"2024-03-27T06:14:18.608966+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:14:18.608966+0000","last_undegraded":"2024-03-27T06:14:18.608966+0000","last_fullsized":"2024-03-27T06:14:18.608966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.808793+0000","last_change":"2024-03-27T06:14:03.526516+0000","last_active":"2024-03-27T06:14:07.808793+0000","last_peered":"2024-03-27T06:14:07.808793+0000","last_clean":"2024-03-27T06:14:07.808793+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:14:07.808793+0000","last_undegraded":"2024-03-27T06:14:07.808793+0000","last_fullsized":"2024-03-27T06:14:07.808793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:42.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.605510+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:14:18.605510+0000","last_peered":"2024-03-27T06:14:18.605510+0000","last_clean":"2024-03-27T06:14:18.605510+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:14:18.605510+0000","last_undegraded":"2024-03-27T06:14:18.605510+0000","last_fullsized":"2024-03-27T06:14:18.605510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708423+0000","last_change":"2024-03-27T06:14:03.522679+0000","last_active":"2024-03-27T06:14:06.708423+0000","last_peered":"2024-03-27T06:14:06.708423+0000","last_clean":"2024-03-27T06:14:06.708423+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:14:06.708423+0000","last_undegraded":"2024-03-27T06:14:06.708423+0000","last_fullsized":"2024-03-27T06:14:06.708423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:13:55.770217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431429+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:14:05.431429+0000","last_peered":"2024-03-27T06:14:05.431429+0000","last_clean":"2024-03-27T06:14:05.431429+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:14:05.431429+0000","last_undegraded":"2024-03-27T06:14:05.431429+0000","last_fullsized":"2024-03-27T06:14:05.431429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.048444+0000","last_change":"2024-03-27T06:14:03.554483+0000","last_active":"2024-03-27T06:14:07.048444+0000","last_peered":"2024-03-27T06:14:07.048444+0000","last_clean":"2024-03-27T06:14:07.048444+0000","last_became_active":"2024-03-27T06:14:03.554183+0000","last_became_peered":"2024-03-27T06:14:03.554183+0000","last_unstale":"2024-03-27T06:14:07.048444+0000","last_undegraded":"2024-03-27T06:14:07.048444+0000","last_fullsized":"2024-03-27T06:14:07.048444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:28:00.308023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708415+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:14:06.708415+0000","last_peered":"2024-03-27T06:14:06.708415+0000","last_clean":"2024-03-27T06:14:06.708415+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:14:06.708415+0000","last_undegraded":"2024-03-27T06:14:06.708415+0000","last_fullsized":"2024-03-27T06:14:06.708415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.462059+0000","last_change":"2024-03-27T06:14:03.526028+0000","last_active":"2024-03-27T06:14:07.462059+0000","last_peered":"2024-03-27T06:14:07.462059+0000","last_clean":"2024-03-27T06:14:07.462059+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:14:07.462059+0000","last_undegraded":"2024-03-27T06:14:07.462059+0000","last_fullsized":"2024-03-27T06:14:07.462059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:05.431591+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:14:05.431591+0000","last_peered":"2024-03-27T06:14:05.431591+0000","last_clean":"2024-03-27T06:14:05.431591+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:14:05.431591+0000","last_undegraded":"2024-03-27T06:14:05.431591+0000","last_fullsized":"2024-03-27T06:14:05.431591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809116+0000","last_change":"2024-03-27T06:14:03.521369+0000","last_active":"2024-03-27T06:14:07.809116+0000","last_peered":"2024-03-27T06:14:07.809116+0000","last_clean":"2024-03-27T06:14:07.809116+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:14:07.809116+0000","last_undegraded":"2024-03-27T06:14:07.809116+0000","last_fullsized":"2024-03-27T06:14:07.809116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:26.303348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613716+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:14:18.613716+0000","last_peered":"2024-03-27T06:14:18.613716+0000","last_clean":"2024-03-27T06:14:18.613716+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:14:18.613716+0000","last_undegraded":"2024-03-27T06:14:18.613716+0000","last_fullsized":"2024-03-27T06:14:18.613716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046680+0000","last_change":"2024-03-27T06:14:03.529137+0000","last_active":"2024-03-27T06:14:07.046680+0000","last_peered":"2024-03-27T06:14:07.046680+0000","last_clean":"2024-03-27T06:14:07.046680+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:14:07.046680+0000","last_undegraded":"2024-03-27T06:14:07.046680+0000","last_fullsized":"2024-03-27T06:14:07.046680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.581236+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:14:13.581236+0000","last_peered":"2024-03-27T06:14:13.581236+0000","last_clean":"2024-03-27T06:14:13.581236+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:14:13.581236+0000","last_undegraded":"2024-03-27T06:14:13.581236+0000","last_fullsized":"2024-03-27T06:14:13.581236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046583+0000","last_change":"2024-03-27T06:14:03.528955+0000","last_active":"2024-03-27T06:14:07.046583+0000","last_peered":"2024-03-27T06:14:07.046583+0000","last_clean":"2024-03-27T06:14:07.046583+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:14:07.046583+0000","last_undegraded":"2024-03-27T06:14:07.046583+0000","last_fullsized":"2024-03-27T06:14:07.046583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:30:34.718045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.584347+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:14:13.584347+0000","last_peered":"2024-03-27T06:14:13.584347+0000","last_clean":"2024-03-27T06:14:13.584347+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:14:13.584347+0000","last_undegraded":"2024-03-27T06:14:13.584347+0000","last_fullsized":"2024-03-27T06:14:13.584347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.529546+0000","last_change":"2024-03-27T06:14:03.539237+0000","last_active":"2024-03-27T06:14:04.529546+0000","last_peered":"2024-03-27T06:14:04.529546+0000","last_clean":"2024-03-27T06:14:04.529546+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:14:04.529546+0000","last_undegraded":"2024-03-27T06:14:04.529546+0000","last_fullsized":"2024-03-27T06:14:04.529546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:15:46.520390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.582654+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:14:13.582654+0000","last_peered":"2024-03-27T06:14:13.582654+0000","last_clean":"2024-03-27T06:14:13.582654+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:14:13.582654+0000","last_undegraded":"2024-03-27T06:14:13.582654+0000","last_fullsized":"2024-03-27T06:14:13.582654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.046354+0000","last_change":"2024-03-27T06:14:03.524796+0000","last_active":"2024-03-27T06:14:07.046354+0000","last_peered":"2024-03-27T06:14:07.046354+0000","last_clean":"2024-03-27T06:14:07.046354+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:14:07.046354+0000","last_undegraded":"2024-03-27T06:14:07.046354+0000","last_fullsized":"2024-03-27T06:14:07.046354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:10:44.237785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:07.809031+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:14:07.809031+0000","last_peered":"2024-03-27T06:14:07.809031+0000","last_clean":"2024-03-27T06:14:07.809031+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:14:07.809031+0000","last_undegraded":"2024-03-27T06:14:07.809031+0000","last_fullsized":"2024-03-27T06:14:07.809031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.527616+0000","last_change":"2024-03-27T06:14:03.515535+0000","last_active":"2024-03-27T06:14:04.527616+0000","last_peered":"2024-03-27T06:14:04.527616+0000","last_clean":"2024-03-27T06:14:04.527616+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:14:04.527616+0000","last_undegraded":"2024-03-27T06:14:04.527616+0000","last_fullsized":"2024-03-27T06:14:04.527616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:13.580898+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:14:13.580898+0000","last_peered":"2024-03-27T06:14:13.580898+0000","last_clean":"2024-03-27T06:14:13.580898+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:14:13.580898+0000","last_undegraded":"2024-03-27T06:14:13.580898+0000","last_fullsized":"2024-03-27T06:14:13.580898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530323+0000","last_change":"2024-03-27T06:14:03.533569+0000","last_active":"2024-03-27T06:14:04.530323+0000","last_peered":"2024-03-27T06:14:04.530323+0000","last_clean":"2024-03-27T06:14:04.530323+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:14:04.530323+0000","last_undegraded":"2024-03-27T06:14:04.530323+0000","last_fullsized":"2024-03-27T06:14:04.530323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:56:45.029930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:04.530063+0000","last_change":"2024-03-27T06:14:03.554653+0000","last_active":"2024-03-27T06:14:04.530063+0000","last_peered":"2024-03-27T06:14:04.530063+0000","last_clean":"2024-03-27T06:14:04.530063+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:14:04.530063+0000","last_undegraded":"2024-03-27T06:14:04.530063+0000","last_fullsized":"2024-03-27T06:14:04.530063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:56:59.309798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.525809+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:14:08.525809+0000","last_peered":"2024-03-27T06:14:08.525809+0000","last_clean":"2024-03-27T06:14:08.525809+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:14:08.525809+0000","last_undegraded":"2024-03-27T06:14:08.525809+0000","last_fullsized":"2024-03-27T06:14:08.525809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.802159+0000","last_change":"2024-03-27T06:14:03.554742+0000","last_active":"2024-03-27T06:14:06.802159+0000","last_peered":"2024-03-27T06:14:06.802159+0000","last_clean":"2024-03-27T06:14:06.802159+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:14:06.802159+0000","last_undegraded":"2024-03-27T06:14:06.802159+0000","last_fullsized":"2024-03-27T06:14:06.802159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.052009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613242+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:14:18.613242+0000","last_peered":"2024-03-27T06:14:18.613242+0000","last_clean":"2024-03-27T06:14:18.613242+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:14:18.613242+0000","last_undegraded":"2024-03-27T06:14:18.613242+0000","last_fullsized":"2024-03-27T06:14:18.613242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708697+0000","last_change":"2024-03-27T06:14:03.530465+0000","last_active":"2024-03-27T06:14:06.708697+0000","last_peered":"2024-03-27T06:14:06.708697+0000","last_clean":"2024-03-27T06:14:06.708697+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:14:06.708697+0000","last_undegraded":"2024-03-27T06:14:06.708697+0000","last_fullsized":"2024-03-27T06:14:06.708697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:53:39.092588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:08.515193+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:14:08.515193+0000","last_peered":"2024-03-27T06:14:08.515193+0000","last_clean":"2024-03-27T06:14:08.515193+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:14:08.515193+0000","last_undegraded":"2024-03-27T06:14:08.515193+0000","last_fullsized":"2024-03-27T06:14:08.515193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:06.708688+0000","last_change":"2024-03-27T06:14:03.531391+0000","last_active":"2024-03-27T06:14:06.708688+0000","last_peered":"2024-03-27T06:14:06.708688+0000","last_clean":"2024-03-27T06:14:06.708688+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:14:06.708688+0000","last_undegraded":"2024-03-27T06:14:06.708688+0000","last_fullsized":"2024-03-27T06:14:06.708688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:24:58.813756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T06:14:18.613729+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:14:18.613729+0000","last_peered":"2024-03-27T06:14:18.613729+0000","last_clean":"2024-03-27T06:14:18.613729+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:14:18.613729+0000","last_undegraded":"2024-03-27T06:14:18.613729+0000","last_fullsized":"2024-03-27T06:14:18.613729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5476,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":10952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":666777,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":79039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666133,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:14:36.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:36.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.344+0000 7f0fe1e00640 1 Processor -- start 2024-03-27T06:14:36.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.344+0000 7f0fe1e00640 1 -- start start 2024-03-27T06:14:36.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.345+0000 7f0fe1e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 0x7f0fdc0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.345+0000 7f0fe1e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fdc15b790 con 0x7f0fdc15aa00 2024-03-27T06:14:36.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fe1e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fdc0c12d0 con 0x7f0fdc0c6fa0 2024-03-27T06:14:36.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fe1e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f0fdc0c1080 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fdaa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 0x7f0fdc0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fda000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f0fdc0c6fa0 0x7f0fdc06b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39938/0 (socket says 172.21.15.132:39938) 2024-03-27T06:14:36.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fda000640 1 -- 172.21.15.132:0/3668958760 learned_addr learned my addr 172.21.15.132:0/3668958760 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:36.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fdaa00640 1 -- 172.21.15.132:0/3668958760 >> v1:172.21.15.134:6789/0 conn(0x7f0fdc0c6fa0 legacy=0x7f0fdc06b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fdaa00640 1 -- 172.21.15.132:0/3668958760 >> v1:172.21.15.132:6789/0 conn(0x7f0fdc15aa00 legacy=0x7f0fdc15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fd9600640 1 -- 172.21.15.132:0/3668958760 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873477898 0 0) 0x7f0fdc0c12d0 con 0x7f0fdc0c6fa0 2024-03-27T06:14:36.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.346+0000 7f0fdaa00640 1 -- 172.21.15.132:0/3668958760 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc0c15b0 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.347+0000 7f0fdaa00640 1 --2- 172.21.15.132:0/3668958760 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 0x7f0fdc0c6a60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0097d0 tx=0x7f0fcc02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe184616dee5de6c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fd9600640 1 -- 172.21.15.132:0/3668958760 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc034070 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fd9600640 1 -- 172.21.15.132:0/3668958760 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc030890 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fd9600640 1 -- 172.21.15.132:0/3668958760 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc030c60 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 -- 172.21.15.132:0/3668958760 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 msgr2=0x7f0fdc0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/3668958760 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 0x7f0fdc0c6a60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0097d0 tx=0x7f0fcc02b9d0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 -- 172.21.15.132:0/3668958760 shutdown_connections 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/3668958760 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc0b11e0 0x7f0fdc0c6a60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 -- 172.21.15.132:0/3668958760 >> 172.21.15.132:0/3668958760 conn(0x7f0fdc0cc060 msgr2=0x7f0fdc0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:36.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 -- 172.21.15.132:0/3668958760 shutdown_connections 2024-03-27T06:14:36.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.348+0000 7f0fe1e00640 1 -- 172.21.15.132:0/3668958760 wait complete. 2024-03-27T06:14:36.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.350+0000 7f0fe1e00640 1 Processor -- start 2024-03-27T06:14:36.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.350+0000 7f0fe1e00640 1 -- start start 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fdc0c6fa0 0x7f0fdc0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 0x7f0fdc0afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0b15c0 con 0x7f0fdc0c6fa0 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0c1c00 con 0x7f0fdc0b11e0 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fe1e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f0fdc0c2090 con 0x7f0fdc15aa00 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fda000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fdc0c6fa0 0x7f0fdc0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fdaa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fdb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 0x7f0fdc0afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fdaa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51034/0 (socket says 172.21.15.132:51034) 2024-03-27T06:14:36.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.351+0000 7f0fdaa00640 1 -- 172.21.15.132:0/1462119255 learned_addr learned my addr 172.21.15.132:0/1462119255 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 -- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 msgr2=0x7f0fdc156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 -- 172.21.15.132:0/1462119255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fdc0c6fa0 msgr2=0x7f0fdc0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fdc0c6fa0 0x7f0fdc0cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 -- 172.21.15.132:0/1462119255 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc1527e0 con 0x7f0fdc15aa00 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdaa00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fdb400640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 0x7f0fdc0afa70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c730 tx=0x7f0fd400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4003b70 con 0x7f0fdc15aa00 2024-03-27T06:14:36.345 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd4003d10 con 0x7f0fdc15aa00 2024-03-27T06:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4005310 con 0x7f0fdc15aa00 2024-03-27T06:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc152ac0 con 0x7f0fdc15aa00 2024-03-27T06:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.352+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc0cb890 con 0x7f0fdc15aa00 2024-03-27T06:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.353+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fdc068840 con 0x7f0fdc15aa00 2024-03-27T06:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.354+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0fd40056c0 con 0x7f0fdc15aa00 2024-03-27T06:14:36.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.355+0000 7f0fd3400640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 0x7f0fbc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.355+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0fd408b650 con 0x7f0fdc15aa00 2024-03-27T06:14:36.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.355+0000 7f0fdaa00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 0x7f0fbc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.356+0000 7f0fdaa00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 0x7f0fbc06e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02ba10 tx=0x7f0fcc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.364+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f0fd4059580 con 0x7f0fdc15aa00 2024-03-27T06:14:36.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.568+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fdc0737e0 con 0x7f0fdc15aa00 2024-03-27T06:14:36.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.569+0000 7f0fd3400640 1 -- 172.21.15.132:0/1462119255 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f0fd405d880 con 0x7f0fdc15aa00 2024-03-27T06:14:36.561 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:36.561 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 msgr2=0x7f0fbc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 0x7f0fbc06e9f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc02ba10 tx=0x7f0fcc03b040 comp rx=0 tx=0).stop 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 msgr2=0x7f0fdc0afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 0x7f0fdc0afa70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c730 tx=0x7f0fd400cbf0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 shutdown_connections 2024-03-27T06:14:36.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0fdc15aa00 0x7f0fdc0afa70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fdc0c6fa0 0x7f0fdc0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0fbc06c530 0x7f0fbc06e9f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 --2- 172.21.15.132:0/1462119255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fdc0b11e0 0x7f0fdc156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 >> 172.21.15.132:0/1462119255 conn(0x7f0fdc0cc060 msgr2=0x7f0fdc0c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 shutdown_connections 2024-03-27T06:14:36.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.573+0000 7f0fe1e00640 1 -- 172.21.15.132:0/1462119255 wait complete. 2024-03-27T06:14:36.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:14:36.591 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:14:36.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.801+0000 7fdb9a400640 1 Processor -- start 2024-03-27T06:14:36.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.801+0000 7fdb9a400640 1 -- start start 2024-03-27T06:14:36.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb9a400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdb9415e230 0x7fdb940c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb9a400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdb940c9570 0x7fdb9415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb9a400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb940c6ec0 con 0x7fdb9415eb50 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb9a400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fdb940c1230 con 0x7fdb940c9570 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb9a400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fdb940c7180 con 0x7fdb9415e230 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb93400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdb9415e230 0x7fdb940c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb8be00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdb940c9570 0x7fdb9415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb93400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdb9415e230 0x7fdb940c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51050/0 (socket says 172.21.15.132:51050) 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.803+0000 7fdb93400640 1 -- 172.21.15.132:0/1948675110 learned_addr learned my addr 172.21.15.132:0/1948675110 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.804+0000 7fdb93400640 1 -- 172.21.15.132:0/1948675110 >> v2:172.21.15.132:3300/0 conn(0x7fdb940c9570 msgr2=0x7fdb9415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.804+0000 7fdb93400640 1 --2- 172.21.15.132:0/1948675110 >> v2:172.21.15.132:3300/0 conn(0x7fdb940c9570 0x7fdb9415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.804+0000 7fdb93400640 1 -- 172.21.15.132:0/1948675110 >> v1:172.21.15.132:6789/0 conn(0x7fdb9415eb50 legacy=0x7fdb94157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.804+0000 7fdb93400640 1 -- 172.21.15.132:0/1948675110 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb940c7460 con 0x7fdb9415e230 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.805+0000 7fdb93400640 1 --2- 172.21.15.132:0/1948675110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb9415e230 0x7fdb940c68a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb94070010 tx=0x7fdb8402ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=745ee3a8b1ef0977 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.805+0000 7fdb92a00640 1 -- 172.21.15.132:0/1948675110 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402f860 con 0x7fdb9415e230 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.805+0000 7fdb92a00640 1 -- 172.21.15.132:0/1948675110 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb8402fa00 con 0x7fdb9415e230 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.805+0000 7fdb92a00640 1 -- 172.21.15.132:0/1948675110 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402fdd0 con 0x7fdb9415e230 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 -- 172.21.15.132:0/1948675110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb9415e230 msgr2=0x7fdb940c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 --2- 172.21.15.132:0/1948675110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb9415e230 0x7fdb940c68a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb94070010 tx=0x7fdb8402ba40 comp rx=0 tx=0).stop 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 -- 172.21.15.132:0/1948675110 shutdown_connections 2024-03-27T06:14:36.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 --2- 172.21.15.132:0/1948675110 >> v2:172.21.15.132:3300/0 conn(0x7fdb940c9570 0x7fdb9415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 --2- 172.21.15.132:0/1948675110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb9415e230 0x7fdb940c68a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 -- 172.21.15.132:0/1948675110 >> 172.21.15.132:0/1948675110 conn(0x7fdb940c89a0 msgr2=0x7fdb940c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 -- 172.21.15.132:0/1948675110 shutdown_connections 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.806+0000 7fdb9a400640 1 -- 172.21.15.132:0/1948675110 wait complete. 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.808+0000 7fdb9a400640 1 Processor -- start 2024-03-27T06:14:36.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.808+0000 7fdb9a400640 1 -- start start 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 0x7fdb9414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 0x7fdb9415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94158260 con 0x7fdb9415eb50 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdb940c0ec0 con 0x7fdb940c9570 2024-03-27T06:14:36.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb9a400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdb941583e0 con 0x7fdb9415e230 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51064/0 (socket says 172.21.15.132:51064) 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 -- 172.21.15.132:0/2219066312 learned_addr learned my addr 172.21.15.132:0/2219066312 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb8be00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 0x7fdb9414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93e00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 0x7fdb9415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 -- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 msgr2=0x7fdb9414d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 0x7fdb9414d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 -- 172.21.15.132:0/2219066312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 msgr2=0x7fdb9415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 0x7fdb9415c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93400640 1 -- 172.21.15.132:0/2219066312 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb941f4470 con 0x7fdb940c9570 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb93e00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 0x7fdb9415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:14:36.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.809+0000 7fdb8be00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 0x7fdb9414d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb93400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402ba80 tx=0x7fdb84004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84037c90 con 0x7fdb940c9570 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb84046070 con 0x7fdb940c9570 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9415d930 con 0x7fdb940c9570 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84041640 con 0x7fdb940c9570 2024-03-27T06:14:36.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.810+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9414d7e0 con 0x7fdb940c9570 2024-03-27T06:14:36.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.811+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb94068950 con 0x7fdb940c9570 2024-03-27T06:14:36.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.812+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdb84040070 con 0x7fdb940c9570 2024-03-27T06:14:36.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.813+0000 7fdb8b400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 0x7fdb7006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:36.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.813+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdb840add90 con 0x7fdb940c9570 2024-03-27T06:14:36.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.813+0000 7fdb8be00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 0x7fdb7006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:36.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.814+0000 7fdb8be00640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 0x7fdb7006ea70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb941515b0 tx=0x7fdb7c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:36.820+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fdb84033030 con 0x7fdb940c9570 2024-03-27T06:14:37.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.019+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb940c8320 con 0x7fdb940c9570 2024-03-27T06:14:37.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.019+0000 7fdb8b400640 1 -- 172.21.15.132:0/2219066312 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fdb8407c420 con 0x7fdb940c9570 2024-03-27T06:14:37.011 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 msgr2=0x7fdb7006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 0x7fdb7006ea70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb941515b0 tx=0x7fdb7c009190 comp rx=0 tx=0).stop 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 msgr2=0x7fdb940aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb8402ba80 tx=0x7fdb84004730 comp rx=0 tx=0).stop 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 shutdown_connections 2024-03-27T06:14:37.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb9415eb50 0x7fdb9415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb9415e230 0x7fdb9414d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7006c5b0 0x7fdb7006ea70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 --2- 172.21.15.132:0/2219066312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb940c9570 0x7fdb940aee70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 >> 172.21.15.132:0/2219066312 conn(0x7fdb940c89a0 msgr2=0x7fdb94072fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 shutdown_connections 2024-03-27T06:14:37.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.022+0000 7fdb9a400640 1 -- 172.21.15.132:0/2219066312 wait complete. 2024-03-27T06:14:37.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:14:37.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.243+0000 7fb870a00640 1 Processor -- start 2024-03-27T06:14:37.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.243+0000 7fb870a00640 1 -- start start 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.244+0000 7fb870a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 0x7fb86c15c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.244+0000 7fb870a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb86c152500 0x7fb86c1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.244+0000 7fb870a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb86c071930 con 0x7fb86c15bfa0 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb870a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fb86c072110 con 0x7fb86c152500 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb870a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb86c06c530 con 0x7fb86c15c380 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb863e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 0x7fb86c15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb869e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb86c15bfa0 0x7fb86c0735f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39948/0 (socket says 172.21.15.132:39948) 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb869e00640 1 -- 172.21.15.132:0/3528793130 learned_addr learned my addr 172.21.15.132:0/3528793130 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb86a800640 1 --2- 172.21.15.132:0/3528793130 >> v2:172.21.15.132:3300/0 conn(0x7fb86c152500 0x7fb86c1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb863e00640 1 -- 172.21.15.132:0/3528793130 >> v2:172.21.15.132:3300/0 conn(0x7fb86c152500 msgr2=0x7fb86c1529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb863e00640 1 --2- 172.21.15.132:0/3528793130 >> v2:172.21.15.132:3300/0 conn(0x7fb86c152500 0x7fb86c1529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb863e00640 1 -- 172.21.15.132:0/3528793130 >> v1:172.21.15.134:6789/0 conn(0x7fb86c15bfa0 legacy=0x7fb86c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb863e00640 1 -- 172.21.15.132:0/3528793130 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86c071bd0 con 0x7fb86c15c380 2024-03-27T06:14:37.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.245+0000 7fb869400640 1 -- 172.21.15.132:0/3528793130 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819922326 0 0) 0x7fb86c071930 con 0x7fb86c15bfa0 2024-03-27T06:14:37.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.246+0000 7fb863e00640 1 --2- 172.21.15.132:0/3528793130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 0x7fb86c15c910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e4d988657a81b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.246+0000 7fb869400640 1 -- 172.21.15.132:0/3528793130 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854034070 con 0x7fb86c15c380 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb869400640 1 -- 172.21.15.132:0/3528793130 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb854030800 con 0x7fb86c15c380 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb869400640 1 -- 172.21.15.132:0/3528793130 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854030bd0 con 0x7fb86c15c380 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 -- 172.21.15.132:0/3528793130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 msgr2=0x7fb86c15c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 --2- 172.21.15.132:0/3528793130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 0x7fb86c15c910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402b940 comp rx=0 tx=0).stop 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 -- 172.21.15.132:0/3528793130 shutdown_connections 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 --2- 172.21.15.132:0/3528793130 >> v2:172.21.15.132:3300/0 conn(0x7fb86c152500 0x7fb86c1529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 --2- 172.21.15.132:0/3528793130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15c380 0x7fb86c15c910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 -- 172.21.15.132:0/3528793130 >> 172.21.15.132:0/3528793130 conn(0x7fb86c0c89a0 msgr2=0x7fb86c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 -- 172.21.15.132:0/3528793130 shutdown_connections 2024-03-27T06:14:37.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.247+0000 7fb870a00640 1 -- 172.21.15.132:0/3528793130 wait complete. 2024-03-27T06:14:37.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.249+0000 7fb870a00640 1 Processor -- start 2024-03-27T06:14:37.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.249+0000 7fb870a00640 1 -- start start 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 0x7fb86c1f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb86c15c380 0x7fb86c0c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c071e70 con 0x7fb86c15c380 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c0c97e0 con 0x7fb86c152500 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.250+0000 7fb870a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb86c06ca60 con 0x7fb86c15bfa0 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb863e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 0x7fb86c1f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb863e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:34866/0 (socket says 172.21.15.132:34866) 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb86a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb86c15c380 0x7fb86c0c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb863e00640 1 -- 172.21.15.132:0/2516535023 learned_addr learned my addr 172.21.15.132:0/2516535023 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 -- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 msgr2=0x7fb86c150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 -- 172.21.15.132:0/2516535023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb86c15c380 msgr2=0x7fb86c0c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb86c15c380 0x7fb86c0c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb869e00640 1 -- 172.21.15.132:0/2516535023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86c0c7cc0 con 0x7fb86c152500 2024-03-27T06:14:37.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.251+0000 7fb863e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb869e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 0x7fb86c1f03c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c450 tx=0x7fb85c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c018070 con 0x7fb86c152500 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb85c0074e0 con 0x7fb86c152500 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004a20 con 0x7fb86c152500 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86c0c7fa0 con 0x7fb86c152500 2024-03-27T06:14:37.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.252+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb86c1f3db0 con 0x7fb86c152500 2024-03-27T06:14:37.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.253+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb86c151a00 con 0x7fb86c152500 2024-03-27T06:14:37.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.254+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb85c004bc0 con 0x7fb86c152500 2024-03-27T06:14:37.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.255+0000 7fb862000640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 0x7fb84806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.255+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb85c08a900 con 0x7fb86c152500 2024-03-27T06:14:37.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.255+0000 7fb863e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 0x7fb84806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.256+0000 7fb863e00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 0x7fb84806e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb85402b980 tx=0x7fb85403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.259+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb86c151a00 con 0x7fb86c152500 2024-03-27T06:14:37.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.461+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb86c068950 con 0x7fb86c152500 2024-03-27T06:14:37.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.462+0000 7fb862000640 1 -- 172.21.15.132:0/2516535023 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fb85c08a020 con 0x7fb86c152500 2024-03-27T06:14:37.453 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 msgr2=0x7fb84806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 0x7fb84806e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb85402b980 tx=0x7fb85403b040 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 msgr2=0x7fb86c1f03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 0x7fb86c1f03c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c450 tx=0x7fb85c00c910 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 shutdown_connections 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb86c15c380 0x7fb86c0c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb86c15bfa0 0x7fb86c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb84806c530 0x7fb84806e9f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 --2- 172.21.15.132:0/2516535023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb86c152500 0x7fb86c1f03c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 >> 172.21.15.132:0/2516535023 conn(0x7fb86c0c89a0 msgr2=0x7fb86c0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:37.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 shutdown_connections 2024-03-27T06:14:37.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.465+0000 7fb870a00640 1 -- 172.21.15.132:0/2516535023 wait complete. 2024-03-27T06:14:37.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:14:37.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.682+0000 7f7aa2a00640 1 Processor -- start 2024-03-27T06:14:37.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.682+0000 7f7aa2a00640 1 -- start start 2024-03-27T06:14:37.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa2a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7a9c154ad0 0x7f7a9c0c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa2a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a9c15ab50 con 0x7f7a9c154eb0 2024-03-27T06:14:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa2a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a9c15ae30 con 0x7f7a9c073040 2024-03-27T06:14:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa2a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f7a9c1441e0 con 0x7f7a9c154ad0 2024-03-27T06:14:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa0a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7a9c073040 0x7f7a9c0c8920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39956/0 (socket says 172.21.15.132:39956) 2024-03-27T06:14:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7aa0a00640 1 -- 172.21.15.132:0/2589997152 learned_addr learned my addr 172.21.15.132:0/2589997152 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:37.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9be00640 1 --2- 172.21.15.132:0/2589997152 >> v2:172.21.15.132:3300/0 conn(0x7f7a9c154ad0 0x7f7a9c0c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9aa00640 1 -- 172.21.15.132:0/2589997152 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599611400 0 0) 0x7f7a9c15ae30 con 0x7f7a9c073040 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9aa00640 1 -- 172.21.15.132:0/2589997152 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a80003520 con 0x7f7a9c073040 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9be00640 1 -- 172.21.15.132:0/2589997152 >> v1:172.21.15.134:6789/0 conn(0x7f7a9c073040 legacy=0x7f7a9c0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9be00640 1 -- 172.21.15.132:0/2589997152 >> v1:172.21.15.132:6789/0 conn(0x7f7a9c154eb0 legacy=0x7f7a9c143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.684+0000 7f7a9be00640 1 -- 172.21.15.132:0/2589997152 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c15b110 con 0x7f7a9c154ad0 2024-03-27T06:14:37.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.686+0000 7f7a9be00640 1 --2- 172.21.15.132:0/2589997152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154ad0 0x7f7a9c0c2b30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d90 tx=0x7f7a8c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854e36738e7b2c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.686+0000 7f7a9aa00640 1 -- 172.21.15.132:0/2589997152 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c039070 con 0x7f7a9c154ad0 2024-03-27T06:14:37.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.686+0000 7f7a9aa00640 1 -- 172.21.15.132:0/2589997152 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c02cc60 con 0x7f7a9c154ad0 2024-03-27T06:14:37.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.686+0000 7f7a9aa00640 1 -- 172.21.15.132:0/2589997152 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c034640 con 0x7f7a9c154ad0 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 -- 172.21.15.132:0/2589997152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154ad0 msgr2=0x7f7a9c0c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/2589997152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154ad0 0x7f7a9c0c2b30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d90 tx=0x7f7a8c02beb0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 -- 172.21.15.132:0/2589997152 shutdown_connections 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/2589997152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154ad0 0x7f7a9c0c2b30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 -- 172.21.15.132:0/2589997152 >> 172.21.15.132:0/2589997152 conn(0x7f7a9c0c9c00 msgr2=0x7f7a9c0ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 -- 172.21.15.132:0/2589997152 shutdown_connections 2024-03-27T06:14:37.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.687+0000 7f7aa2a00640 1 -- 172.21.15.132:0/2589997152 wait complete. 2024-03-27T06:14:37.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.689+0000 7f7aa2a00640 1 Processor -- start 2024-03-27T06:14:37.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.689+0000 7f7aa2a00640 1 -- start start 2024-03-27T06:14:37.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a9c154ad0 0x7f7a9c13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154eb0 0x7f7a9c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c0c7cd0 con 0x7f7a9c154eb0 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c15b760 con 0x7f7a9c073040 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa2a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7a9c0c9030 con 0x7f7a9c154ad0 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7a9be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7a9b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a9c154ad0 0x7f7a9c13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7a9be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51084/0 (socket says 172.21.15.132:51084) 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7a9be00640 1 -- 172.21.15.132:0/21405698 learned_addr learned my addr 172.21.15.132:0/21405698 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.690+0000 7f7aa0a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154eb0 0x7f7a9c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 -- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a9c154ad0 msgr2=0x7f7a9c13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a9c154ad0 0x7f7a9c13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 -- 172.21.15.132:0/21405698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154eb0 msgr2=0x7f7a9c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154eb0 0x7f7a9c0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 -- 172.21.15.132:0/21405698 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c0c4eb0 con 0x7f7a9c073040 2024-03-27T06:14:37.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a9be00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d90 tx=0x7f7a8c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c039050 con 0x7f7a9c073040 2024-03-27T06:14:37.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c03a040 con 0x7f7a9c073040 2024-03-27T06:14:37.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c0b0110 con 0x7f7a9c073040 2024-03-27T06:14:37.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c03e610 con 0x7f7a9c073040 2024-03-27T06:14:37.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.691+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c0b06d0 con 0x7f7a9c073040 2024-03-27T06:14:37.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.693+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9c1545f0 con 0x7f7a9c073040 2024-03-27T06:14:37.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.693+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7a8c047020 con 0x7f7a9c073040 2024-03-27T06:14:37.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.694+0000 7f7a93e00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 0x7f7a7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:37.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.694+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7a8c0ad960 con 0x7f7a9c073040 2024-03-27T06:14:37.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.694+0000 7f7a9b400640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 0x7f7a7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:37.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.695+0000 7f7a9b400640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 0x7f7a7806e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0c7710 tx=0x7f7a84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:37.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.698+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f7a9c1545f0 con 0x7f7a9c073040 2024-03-27T06:14:37.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.915+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7a9c158830 con 0x7f7a9c073040 2024-03-27T06:14:37.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.916+0000 7f7a93e00640 1 -- 172.21.15.132:0/21405698 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f7a8c04e020 con 0x7f7a9c073040 2024-03-27T06:14:37.907 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:14:37.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 msgr2=0x7f7a7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 0x7f7a7806e9f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0c7710 tx=0x7f7a84009040 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 msgr2=0x7f7a9c0b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009d90 tx=0x7f7a8c004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 shutdown_connections 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c154eb0 0x7f7a9c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a9c154ad0 0x7f7a9c13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a7806c530 0x7f7a7806e9f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.918+0000 7f7aa2a00640 1 --2- 172.21.15.132:0/21405698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a9c073040 0x7f7a9c0b2ea0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.919+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 >> 172.21.15.132:0/21405698 conn(0x7f7a9c0c9c00 msgr2=0x7f7a9c0ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.919+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 shutdown_connections 2024-03-27T06:14:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:37.919+0000 7f7aa2a00640 1 -- 172.21.15.132:0/21405698 wait complete. 2024-03-27T06:14:37.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:14:38.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.147+0000 7f122ba00640 1 Processor -- start 2024-03-27T06:14:38.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.147+0000 7f122ba00640 1 -- start start 2024-03-27T06:14:38.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.148+0000 7f122ba00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 0x7f12240c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f122ba00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122415b790 con 0x7f122415aa00 2024-03-27T06:14:38.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f122ba00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12240c12d0 con 0x7f12240c6fa0 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f122ba00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f12240c1080 con 0x7f12240b11e0 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f1229000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 0x7f12240c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f121be00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f12240c6fa0 0x7f122406b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39962/0 (socket says 172.21.15.132:39962) 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f121be00640 1 -- 172.21.15.132:0/2073471428 learned_addr learned my addr 172.21.15.132:0/2073471428 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f1223e00640 1 -- 172.21.15.132:0/2073471428 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693850305 0 0) 0x7f12240c12d0 con 0x7f12240c6fa0 2024-03-27T06:14:38.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f1229000640 1 -- 172.21.15.132:0/2073471428 >> v1:172.21.15.134:6789/0 conn(0x7f12240c6fa0 legacy=0x7f122406b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f1229000640 1 -- 172.21.15.132:0/2073471428 >> v1:172.21.15.132:6789/0 conn(0x7f122415aa00 legacy=0x7f122415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.149+0000 7f1229000640 1 -- 172.21.15.132:0/2073471428 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12240c15b0 con 0x7f12240b11e0 2024-03-27T06:14:38.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.150+0000 7f1229000640 1 --2- 172.21.15.132:0/2073471428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 0x7f12240c6a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f12140097d0 tx=0x7f121402b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=448c49bd9a4a9630 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f1223e00640 1 -- 172.21.15.132:0/2073471428 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214034070 con 0x7f12240b11e0 2024-03-27T06:14:38.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f1223e00640 1 -- 172.21.15.132:0/2073471428 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214030890 con 0x7f12240b11e0 2024-03-27T06:14:38.143 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f1223e00640 1 -- 172.21.15.132:0/2073471428 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214030c60 con 0x7f12240b11e0 2024-03-27T06:14:38.143 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f122ba00640 1 -- 172.21.15.132:0/2073471428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 msgr2=0x7f12240c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f122ba00640 1 --2- 172.21.15.132:0/2073471428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 0x7f12240c6a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f12140097d0 tx=0x7f121402b9d0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.151+0000 7f122ba00640 1 -- 172.21.15.132:0/2073471428 shutdown_connections 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.152+0000 7f122ba00640 1 --2- 172.21.15.132:0/2073471428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240b11e0 0x7f12240c6a60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.152+0000 7f122ba00640 1 -- 172.21.15.132:0/2073471428 >> 172.21.15.132:0/2073471428 conn(0x7f12240cc060 msgr2=0x7f12240cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.152+0000 7f122ba00640 1 -- 172.21.15.132:0/2073471428 shutdown_connections 2024-03-27T06:14:38.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.152+0000 7f122ba00640 1 -- 172.21.15.132:0/2073471428 wait complete. 2024-03-27T06:14:38.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.154+0000 7f122ba00640 1 Processor -- start 2024-03-27T06:14:38.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.154+0000 7f122ba00640 1 -- start start 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 0x7f12240c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 0x7f12241f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f122415aa00 0x7f1224159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12240b15c0 con 0x7f12240b11e0 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12240c1c00 con 0x7f122415aa00 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f122ba00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f12240c2090 con 0x7f12240c6fa0 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f1229000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 0x7f12240c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f1229000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 0x7f12240c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52968/0 (socket says 172.21.15.132:52968) 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f1229000640 1 -- 172.21.15.132:0/272512220 learned_addr learned my addr 172.21.15.132:0/272512220 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f121be00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 0x7f12241f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.156+0000 7f1229a00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f122415aa00 0x7f1224159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 -- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f122415aa00 msgr2=0x7f1224159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f122415aa00 0x7f1224159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 -- 172.21.15.132:0/272512220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 msgr2=0x7f12240c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 0x7f12240c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 -- 172.21.15.132:0/272512220 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12241f0360 con 0x7f12240c6fa0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f121be00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 0x7f12241f41d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c410 tx=0x7f120c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c0074e0 con 0x7f12240c6fa0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f120c0047e0 con 0x7f12240c6fa0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c004bd0 con 0x7f12240c6fa0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12240af400 con 0x7f12240c6fa0 2024-03-27T06:14:38.150 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.157+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12240af9c0 con 0x7f12240c6fa0 2024-03-27T06:14:38.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.159+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1224068840 con 0x7f12240c6fa0 2024-03-27T06:14:38.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.159+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f120c011070 con 0x7f12240c6fa0 2024-03-27T06:14:38.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.160+0000 7f1221600640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 0x7f120006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.160+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f120c08b130 con 0x7f12240c6fa0 2024-03-27T06:14:38.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.161+0000 7f1229000640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 0x7f120006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.161+0000 7f1229000640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 0x7f120006e9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f121402ba10 tx=0x7f121403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.164+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f120c059100 con 0x7f12240c6fa0 2024-03-27T06:14:38.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.399+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12240737e0 con 0x7f12240c6fa0 2024-03-27T06:14:38.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.400+0000 7f1221600640 1 -- 172.21.15.132:0/272512220 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7f120c05d400 con 0x7f12240c6fa0 2024-03-27T06:14:38.392 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:38.392 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:14:38.392 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:14:38.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 msgr2=0x7f120006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 0x7f120006e9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f121402ba10 tx=0x7f121403b040 comp rx=0 tx=0).stop 2024-03-27T06:14:38.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 msgr2=0x7f12241f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 0x7f12241f41d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c410 tx=0x7f120c00c8d0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 shutdown_connections 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f120006c530 0x7f120006e9f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f122415aa00 0x7f1224159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12240c6fa0 0x7f12241f41d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 --2- 172.21.15.132:0/272512220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12240b11e0 0x7f12240c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.402+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 >> 172.21.15.132:0/272512220 conn(0x7f12240cc060 msgr2=0x7f12241586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.403+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 shutdown_connections 2024-03-27T06:14:38.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.403+0000 7f122ba00640 1 -- 172.21.15.132:0/272512220 wait complete. 2024-03-27T06:14:38.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:14:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.617+0000 7fb90f400640 1 Processor -- start 2024-03-27T06:14:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.617+0000 7fb90f400640 1 -- start start 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb90806c8a0 0x7fb9080c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb90806cc80 0x7fb9080c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb9080c77f0 0x7fb908154890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fb9080cc210 con 0x7fb90806cc80 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fb9080cdad0 con 0x7fb90806c8a0 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90f400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb90815b690 con 0x7fb9080c77f0 2024-03-27T06:14:38.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb90806c8a0 0x7fb9080c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb907e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb90806cc80 0x7fb9080c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb90806c8a0 0x7fb9080c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51114/0 (socket says 172.21.15.132:51114) 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb9080c77f0 0x7fb908154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 -- 172.21.15.132:0/2213562736 learned_addr learned my addr 172.21.15.132:0/2213562736 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 -- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb9080c77f0 msgr2=0x7fb908154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 --2- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb9080c77f0 0x7fb908154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 -- 172.21.15.132:0/2213562736 >> v2:172.21.15.132:3300/0 conn(0x7fb90806cc80 msgr2=0x7fb9080c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 --2- 172.21.15.132:0/2213562736 >> v2:172.21.15.132:3300/0 conn(0x7fb90806cc80 0x7fb9080c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.619+0000 7fb90ca00640 1 -- 172.21.15.132:0/2213562736 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90815b970 con 0x7fb90806c8a0 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.620+0000 7fb90ca00640 1 --2- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806c8a0 0x7fb9080c8320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb90808db00 tx=0x7fb8fc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=344e33c42318fb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb907400640 1 -- 172.21.15.132:0/2213562736 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc02c9e0 con 0x7fb90806c8a0 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb907400640 1 -- 172.21.15.132:0/2213562736 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8fc02cb80 con 0x7fb90806c8a0 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb907400640 1 -- 172.21.15.132:0/2213562736 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0343f0 con 0x7fb90806c8a0 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 -- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806c8a0 msgr2=0x7fb9080c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 --2- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806c8a0 0x7fb9080c8320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb90808db00 tx=0x7fb8fc02be40 comp rx=0 tx=0).stop 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 -- 172.21.15.132:0/2213562736 shutdown_connections 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 --2- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb9080c77f0 0x7fb908154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 --2- 172.21.15.132:0/2213562736 >> v2:172.21.15.132:3300/0 conn(0x7fb90806cc80 0x7fb9080c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 --2- 172.21.15.132:0/2213562736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806c8a0 0x7fb9080c8320 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.621+0000 7fb90f400640 1 -- 172.21.15.132:0/2213562736 >> 172.21.15.132:0/2213562736 conn(0x7fb9080ccbe0 msgr2=0x7fb9080ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.622+0000 7fb90f400640 1 -- 172.21.15.132:0/2213562736 shutdown_connections 2024-03-27T06:14:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.622+0000 7fb90f400640 1 -- 172.21.15.132:0/2213562736 wait complete. 2024-03-27T06:14:38.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.624+0000 7fb90f400640 1 Processor -- start 2024-03-27T06:14:38.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.624+0000 7fb90f400640 1 -- start start 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 0x7fb908157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9080c77f0 0x7fb90814f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb90806d440 con 0x7fb9080c77f0 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb90806d2c0 con 0x7fb90806cc80 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90f400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb9080cd700 con 0x7fb90806c8a0 2024-03-27T06:14:38.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb907e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90ca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 0x7fb908157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb907e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51124/0 (socket says 172.21.15.132:51124) 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb907e00640 1 -- 172.21.15.132:0/48405863 learned_addr learned my addr 172.21.15.132:0/48405863 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.625+0000 7fb90d400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9080c77f0 0x7fb90814f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 -- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 msgr2=0x7fb9081f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 -- 172.21.15.132:0/48405863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9080c77f0 msgr2=0x7fb90814f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9080c77f0 0x7fb90814f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 -- 172.21.15.132:0/48405863 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90814f6c0 con 0x7fb90806c8a0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb907e00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90ca00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 0x7fb908157590 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9080cc390 tx=0x7fb8fc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc043030 con 0x7fb90806c8a0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8fc03e450 con 0x7fb90806c8a0 2024-03-27T06:14:38.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb908156210 con 0x7fb90806c8a0 2024-03-27T06:14:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc009510 con 0x7fb90806c8a0 2024-03-27T06:14:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.626+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9081567d0 con 0x7fb90806c8a0 2024-03-27T06:14:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.628+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb908068840 con 0x7fb90806c8a0 2024-03-27T06:14:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.628+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb8fc046070 con 0x7fb90806c8a0 2024-03-27T06:14:38.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.629+0000 7fb904c00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 0x7fb8e806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:14:38.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.629+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb8fc0ad8d0 con 0x7fb90806c8a0 2024-03-27T06:14:38.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.629+0000 7fb907e00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 0x7fb8e806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:14:38.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.630+0000 7fb907e00640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 0x7fb8e806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9080c81a0 tx=0x7fb8f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:14:38.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.633+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb8fc04d070 con 0x7fb90806c8a0 2024-03-27T06:14:38.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.984+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9080b60d0 con 0x7fb90806c8a0 2024-03-27T06:14:38.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.985+0000 7fb904c00640 1 -- 172.21.15.132:0/48405863 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fb8fc07b800 con 0x7fb90806c8a0 2024-03-27T06:14:38.977 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:14:38.978 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:14:04.518306+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:14:38.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 msgr2=0x7fb8e806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 0x7fb8e806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9080c81a0 tx=0x7fb8f4009190 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 msgr2=0x7fb908157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 0x7fb908157590 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9080cc390 tx=0x7fb8fc004a40 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 shutdown_connections 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9080c77f0 0x7fb90814f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb8e806c530 0x7fb8e806e9f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb90806cc80 0x7fb9081f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 --2- 172.21.15.132:0/48405863 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb90806c8a0 0x7fb908157590 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.988+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 >> 172.21.15.132:0/48405863 conn(0x7fb9080ccbe0 msgr2=0x7fb90815a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.989+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 shutdown_connections 2024-03-27T06:14:38.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:14:38.989+0000 7fb90f400640 1 -- 172.21.15.132:0/48405863 wait complete. 2024-03-27T06:14:38.999 DEBUG:tasks.mds_thrash:Yielding 2024-03-27T06:14:39.000 INFO:teuthology.run_tasks:Running task workunit... 2024-03-27T06:14:39.013 INFO:tasks.workunit:Pulling workunits from ref 155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:14:39.013 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-27T06:14:39.014 INFO:tasks.workunit:timeout=6h 2024-03-27T06:14:39.014 INFO:tasks.workunit:cleanup=True 2024-03-27T06:14:39.014 DEBUG:teuthology.orchestra.run.smithi132:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:39.015 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-03-27T06:14:39.015 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 25 secs before thrashing 2024-03-27T06:14:39.067 INFO:teuthology.orchestra.run.smithi132.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 2ah/42d Inode: 1 Links: 2 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-27 06:14:08.509141095 +0000 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-27 06:14:29.569122045 +0000 2024-03-27T06:14:39.068 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-27 06:14:08.509141095 +0000 2024-03-27T06:14:39.068 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-27T06:14:39.069 DEBUG:teuthology.orchestra.run.smithi132:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-27T06:14:39.151 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 155268c4e432a12433aa833f174f9fe3b1016ae0 2024-03-27T06:14:39.201 INFO:tasks.workunit.client.0.smithi132.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-27T06:15:03.095 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:15:03.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:15:04.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:04.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.239+0000 7fcde8000640 1 Processor -- start 2024-03-27T06:15:04.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.240+0000 7fcde8000640 1 -- start start 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde8000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde8000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fcde0073550 0x7fcde0071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde8000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde006b630 con 0x7fcde0073970 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde8000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fcde0152d30 con 0x7fcde0073550 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde8000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fcde006b8d0 con 0x7fcde0152950 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:60670/0 (socket says 172.21.15.132:60670) 2024-03-27T06:15:04.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde5600640 1 -- 172.21.15.132:0/4151545091 learned_addr learned my addr 172.21.15.132:0/4151545091 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:04.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde6000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcde0073970 0x7fcde006cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34310/0 (socket says 172.21.15.132:34310) 2024-03-27T06:15:04.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.241+0000 7fcde4c00640 1 --2- 172.21.15.132:0/4151545091 >> v2:172.21.15.132:3300/0 conn(0x7fcde0073550 0x7fcde0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.242+0000 7fcde5600640 1 -- 172.21.15.132:0/4151545091 >> v2:172.21.15.132:3300/0 conn(0x7fcde0073550 msgr2=0x7fcde0071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.242+0000 7fcde5600640 1 --2- 172.21.15.132:0/4151545091 >> v2:172.21.15.132:3300/0 conn(0x7fcde0073550 0x7fcde0071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.242+0000 7fcde5600640 1 -- 172.21.15.132:0/4151545091 >> v1:172.21.15.134:6789/0 conn(0x7fcde0073970 legacy=0x7fcde006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.242+0000 7fcddfe00640 1 -- 172.21.15.132:0/4151545091 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191145913 0 0) 0x7fcde006b630 con 0x7fcde0073970 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.242+0000 7fcde5600640 1 -- 172.21.15.132:0/4151545091 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde006bbb0 con 0x7fcde0152950 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.243+0000 7fcde5600640 1 --2- 172.21.15.132:0/4151545091 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009870 tx=0x7fcdd402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a199814343875951 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.243+0000 7fcddfe00640 1 -- 172.21.15.132:0/4151545091 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4034070 con 0x7fcde0152950 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.243+0000 7fcddfe00640 1 -- 172.21.15.132:0/4151545091 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd4030840 con 0x7fcde0152950 2024-03-27T06:15:04.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.243+0000 7fcddfe00640 1 -- 172.21.15.132:0/4151545091 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4030c10 con 0x7fcde0152950 2024-03-27T06:15:04.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 -- 172.21.15.132:0/4151545091 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 msgr2=0x7fcde01590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 --2- 172.21.15.132:0/4151545091 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009870 tx=0x7fcdd402b980 comp rx=0 tx=0).stop 2024-03-27T06:15:04.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 -- 172.21.15.132:0/4151545091 shutdown_connections 2024-03-27T06:15:04.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 --2- 172.21.15.132:0/4151545091 >> v2:172.21.15.132:3300/0 conn(0x7fcde0073550 0x7fcde0071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 --2- 172.21.15.132:0/4151545091 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0152950 0x7fcde01590a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 -- 172.21.15.132:0/4151545091 >> 172.21.15.132:0/4151545091 conn(0x7fcde00cc060 msgr2=0x7fcde00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:04.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 -- 172.21.15.132:0/4151545091 shutdown_connections 2024-03-27T06:15:04.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.244+0000 7fcde8000640 1 -- 172.21.15.132:0/4151545091 wait complete. 2024-03-27T06:15:04.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.247+0000 7fcde8000640 1 Processor -- start 2024-03-27T06:15:04.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.247+0000 7fcde8000640 1 -- start start 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.248+0000 7fcde8000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.248+0000 7fcde8000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcde0073970 0x7fcde01508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde8000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde0152950 0x7fcde0150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde8000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcde006c450 con 0x7fcde0152950 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde8000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcde006c170 con 0x7fcde0073970 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde8000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fcde006d660 con 0x7fcde0073550 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:60684/0 (socket says 172.21.15.132:60684) 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 -- 172.21.15.132:0/101835744 learned_addr learned my addr 172.21.15.132:0/101835744 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde4c00640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcde0073970 0x7fcde01508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde6000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde0152950 0x7fcde0150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 -- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcde0073970 msgr2=0x7fcde01508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcde0073970 0x7fcde01508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 -- 172.21.15.132:0/101835744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde0152950 msgr2=0x7fcde0150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde0152950 0x7fcde0150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 -- 172.21.15.132:0/101835744 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde0158540 con 0x7fcde0073550 2024-03-27T06:15:04.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.249+0000 7fcde5600640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4000c00 tx=0x7fcdd40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.250+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4034040 con 0x7fcde0073550 2024-03-27T06:15:04.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.250+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd4039d90 con 0x7fcde0073550 2024-03-27T06:15:04.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.250+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde0158ae0 con 0x7fcde0073550 2024-03-27T06:15:04.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.250+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4043610 con 0x7fcde0073550 2024-03-27T06:15:04.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.250+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde0157370 con 0x7fcde0073550 2024-03-27T06:15:04.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.251+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde0068840 con 0x7fcde0073550 2024-03-27T06:15:04.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.252+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcdd404a060 con 0x7fcde0073550 2024-03-27T06:15:04.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.252+0000 7fcddd600640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 0x7fcdc006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.253+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fcdd40ad750 con 0x7fcde0073550 2024-03-27T06:15:04.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.253+0000 7fcde4c00640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 0x7fcdc006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.253+0000 7fcde4c00640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 0x7fcdc006e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcde015c740 tx=0x7fcdcc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.257+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fcdd403e070 con 0x7fcde0073550 2024-03-27T06:15:04.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.491+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcde0152280 con 0x7fcde0073550 2024-03-27T06:15:04.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.492+0000 7fcddd600640 1 -- 172.21.15.132:0/101835744 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fcdd407bc20 con 0x7fcde0073550 2024-03-27T06:15:04.483 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:04.483 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:04.483 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 msgr2=0x7fcdc006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 0x7fcdc006e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcde015c740 tx=0x7fcdcc009190 comp rx=0 tx=0).stop 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 msgr2=0x7fcde0147070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4000c00 tx=0x7fcdd40046c0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 shutdown_connections 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde0152950 0x7fcde0150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcdc006c530 0x7fcdc006e9f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcde0073970 0x7fcde01508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 --2- 172.21.15.132:0/101835744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcde0073550 0x7fcde0147070 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 >> 172.21.15.132:0/101835744 conn(0x7fcde00cc060 msgr2=0x7fcde006da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:04.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 shutdown_connections 2024-03-27T06:15:04.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.495+0000 7fcde8000640 1 -- 172.21.15.132:0/101835744 wait complete. 2024-03-27T06:15:04.511 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2024-03-27T06:15:04.511 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-27T06:15:04.512 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:15:04.530 INFO:tasks.ceph.mds.c:Stopped 2024-03-27T06:15:04.530 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:04.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.786+0000 7f2a6cc00640 1 Processor -- start 2024-03-27T06:15:04.777 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.786+0000 7f2a6cc00640 1 -- start start 2024-03-27T06:15:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a6cc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2a681549c0 0x7f2a6815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a6cc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2a6806c7a0 0x7f2a6806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a6cc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a68071220 con 0x7f2a6806bf80 2024-03-27T06:15:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a6cc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2a68071d80 con 0x7f2a6806c7a0 2024-03-27T06:15:04.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a6cc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6806d020 con 0x7f2a681549c0 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a67400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2a681549c0 0x7f2a6815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a67e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2a6806bf80 0x7f2a68072210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34318/0 (socket says 172.21.15.132:34318) 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a67e00640 1 -- 172.21.15.132:0/3258508385 learned_addr learned my addr 172.21.15.132:0/3258508385 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.788+0000 7f2a5fe00640 1 --2- 172.21.15.132:0/3258508385 >> v2:172.21.15.132:3300/0 conn(0x7f2a6806c7a0 0x7f2a6806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.789+0000 7f2a67400640 1 -- 172.21.15.132:0/3258508385 >> v2:172.21.15.132:3300/0 conn(0x7f2a6806c7a0 msgr2=0x7f2a6806ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.789+0000 7f2a67400640 1 --2- 172.21.15.132:0/3258508385 >> v2:172.21.15.132:3300/0 conn(0x7f2a6806c7a0 0x7f2a6806ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.789+0000 7f2a67400640 1 -- 172.21.15.132:0/3258508385 >> v1:172.21.15.134:6789/0 conn(0x7f2a6806bf80 legacy=0x7f2a68072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.789+0000 7f2a67400640 1 -- 172.21.15.132:0/3258508385 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6806d300 con 0x7f2a681549c0 2024-03-27T06:15:04.780 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.789+0000 7f2a66a00640 1 -- 172.21.15.132:0/3258508385 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693461499 0 0) 0x7f2a60001490 con 0x7f2a6806bf80 2024-03-27T06:15:04.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.790+0000 7f2a67400640 1 --2- 172.21.15.132:0/3258508385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a681549c0 0x7f2a6815b1f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2a680c9770 tx=0x7f2a5802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f715de179cfc08e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.790+0000 7f2a66a00640 1 -- 172.21.15.132:0/3258508385 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58034070 con 0x7f2a681549c0 2024-03-27T06:15:04.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.790+0000 7f2a66a00640 1 -- 172.21.15.132:0/3258508385 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58030610 con 0x7f2a681549c0 2024-03-27T06:15:04.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.790+0000 7f2a66a00640 1 -- 172.21.15.132:0/3258508385 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a580309e0 con 0x7f2a681549c0 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3258508385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a681549c0 msgr2=0x7f2a6815b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3258508385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a681549c0 0x7f2a6815b1f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2a680c9770 tx=0x7f2a5802b960 comp rx=0 tx=0).stop 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3258508385 shutdown_connections 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3258508385 >> v2:172.21.15.132:3300/0 conn(0x7f2a6806c7a0 0x7f2a6806ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3258508385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a681549c0 0x7f2a6815b1f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3258508385 >> 172.21.15.132:0/3258508385 conn(0x7f2a680cc7a0 msgr2=0x7f2a680c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3258508385 shutdown_connections 2024-03-27T06:15:04.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.791+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3258508385 wait complete. 2024-03-27T06:15:04.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.792+0000 7f2a6cc00640 1 Processor -- start 2024-03-27T06:15:04.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 -- start start 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a6806bf80 0x7f2a681e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a681549c0 0x7f2a6814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a68071a10 con 0x7f2a6806bf80 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2a68071730 con 0x7f2a6806c7a0 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.793+0000 7f2a6cc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6806cbc0 con 0x7f2a681549c0 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:56036/0 (socket says 172.21.15.132:56036) 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 -- 172.21.15.132:0/3222083391 learned_addr learned my addr 172.21.15.132:0/3222083391 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a67e00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a681549c0 0x7f2a6814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a67400640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a6806bf80 0x7f2a681e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 -- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a681549c0 msgr2=0x7f2a6814dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a681549c0 0x7f2a6814dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 -- 172.21.15.132:0/3222083391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a6806bf80 msgr2=0x7f2a681e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:04.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a6806bf80 0x7f2a681e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:04.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 -- 172.21.15.132:0/3222083391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6815b910 con 0x7f2a6806c7a0 2024-03-27T06:15:04.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.794+0000 7f2a5fe00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e8a0 tx=0x7f2a5000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.795+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50017070 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.795+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5000c940 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.795+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000cd30 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.795+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a68152af0 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.795+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6815be90 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.796+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a680c44b0 con 0x7f2a6806c7a0 2024-03-27T06:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.797+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2a5001a070 con 0x7f2a6806c7a0 2024-03-27T06:15:04.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.798+0000 7f2a5f400640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 0x7f2a4406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:04.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.798+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7f2a5008b530 con 0x7f2a6806c7a0 2024-03-27T06:15:04.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.798+0000 7f2a67400640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 0x7f2a4406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:04.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.798+0000 7f2a67400640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 0x7f2a4406ea20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2a58036000 tx=0x7f2a5803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:04.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:04.806+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2a680c44b0 con 0x7f2a6806c7a0 2024-03-27T06:15:05.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.027+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a680686c0 con 0x7f2a6806c7a0 2024-03-27T06:15:05.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.028+0000 7f2a5f400640 1 -- 172.21.15.132:0/3222083391 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7f2a50059440 con 0x7f2a6806c7a0 2024-03-27T06:15:05.019 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:05.019 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:05.019 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:05.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 msgr2=0x7f2a4406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:05.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 0x7f2a4406ea20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2a58036000 tx=0x7f2a5803b040 comp rx=0 tx=0).stop 2024-03-27T06:15:05.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 msgr2=0x7f2a6815cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e8a0 tx=0x7f2a5000ed60 comp rx=0 tx=0).stop 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 shutdown_connections 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a681549c0 0x7f2a6814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a4406c560 0x7f2a4406ea20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a6806c7a0 0x7f2a6815cec0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 --2- 172.21.15.132:0/3222083391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a6806bf80 0x7f2a681e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.030+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 >> 172.21.15.132:0/3222083391 conn(0x7f2a680cc7a0 msgr2=0x7f2a680c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.031+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 shutdown_connections 2024-03-27T06:15:05.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:05.031+0000 7f2a6cc00640 1 -- 172.21.15.132:0/3222083391 wait complete. 2024-03-27T06:15:05.038 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:07.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:07.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.267+0000 7fee7ee00640 1 Processor -- start 2024-03-27T06:15:07.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.267+0000 7fee7ee00640 1 -- start start 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee7ee00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fee78151ad0 0x7fee78157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee7ee00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7806b520 con 0x7fee78073880 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee7ee00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7806b7c0 con 0x7fee781516f0 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee7ee00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fee78072be0 con 0x7fee78151ad0 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee77e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fee781516f0 0x7fee780c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34334/0 (socket says 172.21.15.132:34334) 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.268+0000 7fee77e00640 1 -- 172.21.15.132:0/4278834807 learned_addr learned my addr 172.21.15.132:0/4278834807 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:07.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee6f400640 1 --2- 172.21.15.132:0/4278834807 >> v2:172.21.15.132:3300/0 conn(0x7fee78151ad0 0x7fee78157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:07.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717023040 0 0) 0x7fee7806b7c0 con 0x7fee781516f0 2024-03-27T06:15:07.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee5c003520 con 0x7fee781516f0 2024-03-27T06:15:07.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724961331 0 0) 0x7fee7806b520 con 0x7fee78073880 2024-03-27T06:15:07.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee7806b7c0 con 0x7fee78073880 2024-03-27T06:15:07.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee6f400640 1 -- 172.21.15.132:0/4278834807 >> v1:172.21.15.134:6789/0 conn(0x7fee781516f0 legacy=0x7fee780c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2971285234 0 0) 0x7fee5c003520 con 0x7fee781516f0 2024-03-27T06:15:07.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee6f400640 1 -- 172.21.15.132:0/4278834807 >> v1:172.21.15.132:6789/0 conn(0x7fee78073880 legacy=0x7fee78161e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.269+0000 7fee6f400640 1 -- 172.21.15.132:0/4278834807 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7806baa0 con 0x7fee78151ad0 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.270+0000 7fee6f400640 1 --2- 172.21.15.132:0/4278834807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee78151ad0 0x7fee78157570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee60009720 tx=0x7fee6002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d80947cf327951b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.270+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68001670 con 0x7fee78151ad0 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee60037070 con 0x7fee78151ad0 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee77400640 1 -- 172.21.15.132:0/4278834807 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600317e0 con 0x7fee78151ad0 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 -- 172.21.15.132:0/4278834807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee78151ad0 msgr2=0x7fee78157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 --2- 172.21.15.132:0/4278834807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee78151ad0 0x7fee78157570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee60009720 tx=0x7fee6002b860 comp rx=0 tx=0).stop 2024-03-27T06:15:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 -- 172.21.15.132:0/4278834807 shutdown_connections 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 --2- 172.21.15.132:0/4278834807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee78151ad0 0x7fee78157570 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 -- 172.21.15.132:0/4278834807 >> 172.21.15.132:0/4278834807 conn(0x7fee780cc060 msgr2=0x7fee780cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 -- 172.21.15.132:0/4278834807 shutdown_connections 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.271+0000 7fee7ee00640 1 -- 172.21.15.132:0/4278834807 wait complete. 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.273+0000 7fee7ee00640 1 Processor -- start 2024-03-27T06:15:07.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.273+0000 7fee7ee00640 1 -- start start 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 0x7fee781f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fee78151ad0 0x7fee78151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fee7806bd80 con 0x7fee781516f0 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fee78072870 con 0x7fee78073880 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ee00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fee780c1080 con 0x7fee78151ad0 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 0x7fee781f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee6f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:07.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee7ce00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fee78151ad0 0x7fee78151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee6f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50090/0 (socket says 172.21.15.132:50090) 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.274+0000 7fee6f400640 1 -- 172.21.15.132:0/3545279194 learned_addr learned my addr 172.21.15.132:0/3545279194 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 -- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fee78151ad0 msgr2=0x7fee78151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fee78151ad0 0x7fee78151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 -- 172.21.15.132:0/3545279194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 msgr2=0x7fee780cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 -- 172.21.15.132:0/3545279194 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee78152900 con 0x7fee78073880 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee6f400640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee77e00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 0x7fee781f0350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c750 tx=0x7fee6800cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68018070 con 0x7fee78073880 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee680072f0 con 0x7fee78073880 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee680047e0 con 0x7fee78073880 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee78152be0 con 0x7fee78073880 2024-03-27T06:15:07.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.275+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee78153830 con 0x7fee78073880 2024-03-27T06:15:07.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.276+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee78068840 con 0x7fee78073880 2024-03-27T06:15:07.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.277+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fee6801b070 con 0x7fee78073880 2024-03-27T06:15:07.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.278+0000 7fee74c00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 0x7fee5406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:07.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.278+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fee6808ab70 con 0x7fee78073880 2024-03-27T06:15:07.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.278+0000 7fee6f400640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 0x7fee5406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:07.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.279+0000 7fee6f400640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 0x7fee5406ea20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fee6002b8a0 tx=0x7fee60008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:07.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.287+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fee68058ab0 con 0x7fee78073880 2024-03-27T06:15:07.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.524+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee78153100 con 0x7fee78073880 2024-03-27T06:15:07.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.525+0000 7fee74c00640 1 -- 172.21.15.132:0/3545279194 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fee6805cdb0 con 0x7fee78073880 2024-03-27T06:15:07.516 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:07.516 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:07.517 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:07.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.528+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 msgr2=0x7fee5406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.528+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 0x7fee5406ea20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fee6002b8a0 tx=0x7fee60008030 comp rx=0 tx=0).stop 2024-03-27T06:15:07.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.528+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 msgr2=0x7fee781f0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:07.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.528+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 0x7fee781f0350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c750 tx=0x7fee6800cc10 comp rx=0 tx=0).stop 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 shutdown_connections 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fee78151ad0 0x7fee78151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee781516f0 0x7fee780cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee5406c560 0x7fee5406ea20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 --2- 172.21.15.132:0/3545279194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee78073880 0x7fee781f0350 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 >> 172.21.15.132:0/3545279194 conn(0x7fee780cc060 msgr2=0x7fee7806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 shutdown_connections 2024-03-27T06:15:07.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:07.529+0000 7fee7ee00640 1 -- 172.21.15.132:0/3545279194 wait complete. 2024-03-27T06:15:07.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:09.540 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:09.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.767+0000 7fa203e00640 1 Processor -- start 2024-03-27T06:15:09.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.767+0000 7fa203e00640 1 -- start start 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.768+0000 7fa203e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.768+0000 7fa203e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa204156e50 con 0x7fa204072590 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.768+0000 7fa203e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa204157110 con 0x7fa2040b1d30 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.768+0000 7fa203e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa2041573f0 con 0x7fa204160600 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa203400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa202a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa2040b1d30 0x7fa2040c5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34336/0 (socket says 172.21.15.132:34336) 2024-03-27T06:15:09.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa202a00640 1 -- 172.21.15.132:0/2939559101 learned_addr learned my addr 172.21.15.132:0/2939559101 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:09.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa201600640 1 -- 172.21.15.132:0/2939559101 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581966254 0 0) 0x7fa204157110 con 0x7fa2040b1d30 2024-03-27T06:15:09.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa203400640 1 -- 172.21.15.132:0/2939559101 >> v1:172.21.15.134:6789/0 conn(0x7fa2040b1d30 legacy=0x7fa2040c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:09.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa203400640 1 -- 172.21.15.132:0/2939559101 >> v1:172.21.15.132:6789/0 conn(0x7fa204072590 legacy=0x7fa2040729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:09.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.769+0000 7fa203400640 1 -- 172.21.15.132:0/2939559101 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2041576d0 con 0x7fa204160600 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.770+0000 7fa203400640 1 --2- 172.21.15.132:0/2939559101 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041629f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009dc0 tx=0x7fa1fc02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14e0dc15999ef5c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.770+0000 7fa201600640 1 -- 172.21.15.132:0/2939559101 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc039070 con 0x7fa204160600 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.770+0000 7fa201600640 1 -- 172.21.15.132:0/2939559101 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1fc02cd60 con 0x7fa204160600 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.770+0000 7fa201600640 1 -- 172.21.15.132:0/2939559101 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc034660 con 0x7fa204160600 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 -- 172.21.15.132:0/2939559101 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 msgr2=0x7fa2041629f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:09.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 --2- 172.21.15.132:0/2939559101 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041629f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009dc0 tx=0x7fa1fc02bfb0 comp rx=0 tx=0).stop 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 -- 172.21.15.132:0/2939559101 shutdown_connections 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 --2- 172.21.15.132:0/2939559101 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041629f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 -- 172.21.15.132:0/2939559101 >> 172.21.15.132:0/2939559101 conn(0x7fa2040cc7a0 msgr2=0x7fa2040c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 -- 172.21.15.132:0/2939559101 shutdown_connections 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.771+0000 7fa203e00640 1 -- 172.21.15.132:0/2939559101 wait complete. 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.773+0000 7fa203e00640 1 Processor -- start 2024-03-27T06:15:09.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.773+0000 7fa203e00640 1 -- start start 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa204072590 0x7fa2041f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2040b1d30 0x7fa20406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa2040c75e0 con 0x7fa2040b1d30 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa20406ca90 con 0x7fa204072590 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa204146000 con 0x7fa204160600 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa202a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa204072590 0x7fa2041f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:60708/0 (socket says 172.21.15.132:60708) 2024-03-27T06:15:09.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 -- 172.21.15.132:0/94430684 learned_addr learned my addr 172.21.15.132:0/94430684 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa202000640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2040b1d30 0x7fa20406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 -- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa204072590 msgr2=0x7fa2041f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa204072590 0x7fa2041f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 -- 172.21.15.132:0/94430684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2040b1d30 msgr2=0x7fa20406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2040b1d30 0x7fa20406e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:09.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.774+0000 7fa203400640 1 -- 172.21.15.132:0/94430684 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2041f4660 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa203400640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc037040 tx=0x7fa1fc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc039050 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1fc045070 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20414f650 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc03e830 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.775+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2041f3920 con 0x7fa204160600 2024-03-27T06:15:09.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.776+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2040686c0 con 0x7fa204160600 2024-03-27T06:15:09.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.777+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa1fc03e9d0 con 0x7fa204160600 2024-03-27T06:15:09.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.778+0000 7fa1fb400640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 0x7fa1e406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:09.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.778+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fa1fc0ae570 con 0x7fa204160600 2024-03-27T06:15:09.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.778+0000 7fa202a00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 0x7fa1e406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:09.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.779+0000 7fa202a00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 0x7fa1e406e9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4007f70 tx=0x7fa1f4007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:09.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:09.782+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa1fc07c5a0 con 0x7fa204160600 2024-03-27T06:15:10.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.022+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2040711a0 con 0x7fa204160600 2024-03-27T06:15:10.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.023+0000 7fa1fb400640 1 -- 172.21.15.132:0/94430684 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fa1fc0808a0 con 0x7fa204160600 2024-03-27T06:15:10.014 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:10.014 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:10.014 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:10.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 msgr2=0x7fa1e406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 0x7fa1e406e9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4007f70 tx=0x7fa1f4007f00 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 msgr2=0x7fa2041f08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc037040 tx=0x7fa1fc004a40 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 shutdown_connections 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa204160600 0x7fa2041f08e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2040b1d30 0x7fa20406e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1e406c530 0x7fa1e406e9f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 --2- 172.21.15.132:0/94430684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa204072590 0x7fa2041f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 >> 172.21.15.132:0/94430684 conn(0x7fa2040cc7a0 msgr2=0x7fa20406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 shutdown_connections 2024-03-27T06:15:10.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:10.026+0000 7fa203e00640 1 -- 172.21.15.132:0/94430684 wait complete. 2024-03-27T06:15:10.035 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:12.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.268+0000 7fe8a3000640 1 Processor -- start 2024-03-27T06:15:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.268+0000 7fe8a3000640 1 -- start start 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a3000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a3000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 0x7fe89c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a3000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe89c15b790 con 0x7fe89c0c6fa0 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a3000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fe89c0c1080 con 0x7fe89c15aa00 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a3000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe89c0c1c00 con 0x7fe89c0b11e0 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe89be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe8a1000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 0x7fe89c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe89be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59450/0 (socket says 172.21.15.132:59450) 2024-03-27T06:15:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.270+0000 7fe89be00640 1 -- 172.21.15.132:0/800058155 learned_addr learned my addr 172.21.15.132:0/800058155 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.271+0000 7fe89be00640 1 -- 172.21.15.132:0/800058155 >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 msgr2=0x7fe89c15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.271+0000 7fe89be00640 1 --2- 172.21.15.132:0/800058155 >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 0x7fe89c15ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.271+0000 7fe89be00640 1 -- 172.21.15.132:0/800058155 >> v1:172.21.15.132:6789/0 conn(0x7fe89c0c6fa0 legacy=0x7fe89c06b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.271+0000 7fe89be00640 1 -- 172.21.15.132:0/800058155 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c0c12d0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.271+0000 7fe8a1000640 1 --2- 172.21.15.132:0/800058155 >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 0x7fe89c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.272+0000 7fe89be00640 1 --2- 172.21.15.132:0/800058155 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009870 tx=0x7fe88c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a27cadfdc515b7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.272+0000 7fe89aa00640 1 -- 172.21.15.132:0/800058155 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c02f830 con 0x7fe89c0b11e0 2024-03-27T06:15:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.272+0000 7fe89aa00640 1 -- 172.21.15.132:0/800058155 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88c02f9d0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.272+0000 7fe89aa00640 1 -- 172.21.15.132:0/800058155 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c02fda0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 -- 172.21.15.132:0/800058155 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 msgr2=0x7fe89c0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 --2- 172.21.15.132:0/800058155 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009870 tx=0x7fe88c02ba10 comp rx=0 tx=0).stop 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 -- 172.21.15.132:0/800058155 shutdown_connections 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 --2- 172.21.15.132:0/800058155 >> v2:172.21.15.132:3300/0 conn(0x7fe89c15aa00 0x7fe89c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 --2- 172.21.15.132:0/800058155 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0b11e0 0x7fe89c0c6a60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 -- 172.21.15.132:0/800058155 >> 172.21.15.132:0/800058155 conn(0x7fe89c0cc060 msgr2=0x7fe89c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 -- 172.21.15.132:0/800058155 shutdown_connections 2024-03-27T06:15:12.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.273+0000 7fe8a3000640 1 -- 172.21.15.132:0/800058155 wait complete. 2024-03-27T06:15:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.275+0000 7fe8a3000640 1 Processor -- start 2024-03-27T06:15:12.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.275+0000 7fe8a3000640 1 -- start start 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0c6fa0 0x7fe89c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 0x7fe89c0afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c0b15c0 con 0x7fe89c15aa00 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c0c1890 con 0x7fe89c0b11e0 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a3000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe89c0c2090 con 0x7fe89c0c6fa0 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:40692/0 (socket says 172.21.15.132:40692) 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 -- 172.21.15.132:0/3638484940 learned_addr learned my addr 172.21.15.132:0/3638484940 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89b400640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0c6fa0 0x7fe89c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a1000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 0x7fe89c0afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 -- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0c6fa0 msgr2=0x7fe89c156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0c6fa0 0x7fe89c156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 -- 172.21.15.132:0/3638484940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 msgr2=0x7fe89c0afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 0x7fe89c0afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe89be00640 1 -- 172.21.15.132:0/3638484940 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c1522d0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.276+0000 7fe8a1000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 0x7fe89c0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:15:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe89be00640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000c00 tx=0x7fe88c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c037c40 con 0x7fe89c0b11e0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88c046070 con 0x7fe89c0b11e0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c041400 con 0x7fe89c0b11e0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89c1f1a80 con 0x7fe89c0b11e0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.277+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89c1f0f00 con 0x7fe89c0b11e0 2024-03-27T06:15:12.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.278+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe89c068840 con 0x7fe89c0b11e0 2024-03-27T06:15:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.279+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe88c040070 con 0x7fe89c0b11e0 2024-03-27T06:15:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.280+0000 7fe893e00640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 0x7fe87c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:12.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.280+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fe88c0adcc0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.280+0000 7fe89b400640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 0x7fe87c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.281+0000 7fe89b400640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 0x7fe87c06e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0b0550 tx=0x7fe888009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:12.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.288+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe88c0b4080 con 0x7fe89c0b11e0 2024-03-27T06:15:12.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.526+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe89c0c3230 con 0x7fe89c0b11e0 2024-03-27T06:15:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.527+0000 7fe893e00640 1 -- 172.21.15.132:0/3638484940 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fe89c1522d0 con 0x7fe89c0b11e0 2024-03-27T06:15:12.519 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:12.519 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:12.520 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 msgr2=0x7fe87c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 0x7fe87c06e9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0b0550 tx=0x7fe888009040 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 msgr2=0x7fe89c151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000c00 tx=0x7fe88c004730 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 shutdown_connections 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe89c15aa00 0x7fe89c0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe89c0c6fa0 0x7fe89c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe87c06c530 0x7fe87c06e9f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 --2- 172.21.15.132:0/3638484940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe89c0b11e0 0x7fe89c151d90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 >> 172.21.15.132:0/3638484940 conn(0x7fe89c0cc060 msgr2=0x7fe89c153b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 shutdown_connections 2024-03-27T06:15:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:12.530+0000 7fe8a3000640 1 -- 172.21.15.132:0/3638484940 wait complete. 2024-03-27T06:15:12.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:14.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:14.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.766+0000 7fd8ca000640 1 Processor -- start 2024-03-27T06:15:14.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.766+0000 7fd8ca000640 1 -- start start 2024-03-27T06:15:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.767+0000 7fd8ca000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd8c41574f0 0x7fd8c40b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.767+0000 7fd8ca000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c4157ad0 con 0x7fd8c4151830 2024-03-27T06:15:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.767+0000 7fd8ca000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c4157db0 con 0x7fd8c40b1c90 2024-03-27T06:15:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.767+0000 7fd8ca000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c4071290 con 0x7fd8c41574f0 2024-03-27T06:15:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c3e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd8c41574f0 0x7fd8c40b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c8c00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd8c4151830 0x7fd8c40c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39340/0 (socket says 172.21.15.132:39340) 2024-03-27T06:15:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c8c00640 1 -- 172.21.15.132:0/1915004689 learned_addr learned my addr 172.21.15.132:0/1915004689 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c3e00640 1 -- 172.21.15.132:0/1915004689 >> v1:172.21.15.134:6789/0 conn(0x7fd8c40b1c90 legacy=0x7fd8c40730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c3e00640 1 -- 172.21.15.132:0/1915004689 >> v1:172.21.15.132:6789/0 conn(0x7fd8c4151830 legacy=0x7fd8c40c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c3400640 1 -- 172.21.15.132:0/1915004689 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477959306 0 0) 0x7fd8c4157db0 con 0x7fd8c40b1c90 2024-03-27T06:15:14.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.768+0000 7fd8c3e00640 1 -- 172.21.15.132:0/1915004689 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c4158090 con 0x7fd8c41574f0 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.769+0000 7fd8c3e00640 1 --2- 172.21.15.132:0/1915004689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c41574f0 0x7fd8c40b1750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097a0 tx=0x7fd8b002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ee8e0495a319f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8c3400640 1 -- 172.21.15.132:0/1915004689 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0038070 con 0x7fd8c41574f0 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8c3400640 1 -- 172.21.15.132:0/1915004689 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0031700 con 0x7fd8c41574f0 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8c3400640 1 -- 172.21.15.132:0/1915004689 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0031ad0 con 0x7fd8c41574f0 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 -- 172.21.15.132:0/1915004689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c41574f0 msgr2=0x7fd8c40b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 --2- 172.21.15.132:0/1915004689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c41574f0 0x7fd8c40b1750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097a0 tx=0x7fd8b002b820 comp rx=0 tx=0).stop 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 -- 172.21.15.132:0/1915004689 shutdown_connections 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 --2- 172.21.15.132:0/1915004689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c41574f0 0x7fd8c40b1750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 -- 172.21.15.132:0/1915004689 >> 172.21.15.132:0/1915004689 conn(0x7fd8c40cc7a0 msgr2=0x7fd8c40c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 -- 172.21.15.132:0/1915004689 shutdown_connections 2024-03-27T06:15:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.770+0000 7fd8ca000640 1 -- 172.21.15.132:0/1915004689 wait complete. 2024-03-27T06:15:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.772+0000 7fd8ca000640 1 Processor -- start 2024-03-27T06:15:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.772+0000 7fd8ca000640 1 -- start start 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c40b1c90 0x7fd8c40c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd8c41574f0 0x7fd8c4156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4072b90 con 0x7fd8c40b1c90 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c40728b0 con 0x7fd8c4151830 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8ca000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c4151c10 con 0x7fd8c41574f0 2024-03-27T06:15:14.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8c3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:40694/0 (socket says 172.21.15.132:40694) 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8c3e00640 1 -- 172.21.15.132:0/507657634 learned_addr learned my addr 172.21.15.132:0/507657634 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8c8c00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c40b1c90 0x7fd8c40c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.773+0000 7fd8c9600640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd8c41574f0 0x7fd8c4156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 -- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd8c41574f0 msgr2=0x7fd8c4156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd8c41574f0 0x7fd8c4156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 -- 172.21.15.132:0/507657634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c40b1c90 msgr2=0x7fd8c40c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c40b1c90 0x7fd8c40c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 -- 172.21.15.132:0/507657634 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c414f2c0 con 0x7fd8c4151830 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c3e00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097a0 tx=0x7fd8b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0038040 con 0x7fd8c4151830 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0004030 con 0x7fd8c4151830 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0004400 con 0x7fd8c4151830 2024-03-27T06:15:14.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.774+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c414f5a0 con 0x7fd8c4151830 2024-03-27T06:15:14.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.775+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c414fb60 con 0x7fd8c4151830 2024-03-27T06:15:14.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.776+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c414fda0 con 0x7fd8c4151830 2024-03-27T06:15:14.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.776+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd8b00339b0 con 0x7fd8c4151830 2024-03-27T06:15:14.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.777+0000 7fd8c0c00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 0x7fd8a406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:14.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.777+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fd8b00ad3f0 con 0x7fd8c4151830 2024-03-27T06:15:14.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.777+0000 7fd8c8c00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 0x7fd8a406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:14.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.778+0000 7fd8c8c00640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 0x7fd8a406e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000f60 tx=0x7fd8b8007e00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:14.782+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd8b0043020 con 0x7fd8c4151830 2024-03-27T06:15:15.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.018+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c40c7510 con 0x7fd8c4151830 2024-03-27T06:15:15.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.019+0000 7fd8c0c00640 1 -- 172.21.15.132:0/507657634 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fd8c40c7510 con 0x7fd8c4151830 2024-03-27T06:15:15.011 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:15.012 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:15.012 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 msgr2=0x7fd8a406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 0x7fd8a406e9f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000f60 tx=0x7fd8b8007e00 comp rx=0 tx=0).stop 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 msgr2=0x7fd8c40c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097a0 tx=0x7fd8b00046c0 comp rx=0 tx=0).stop 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 shutdown_connections 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd8c41574f0 0x7fd8c4156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:15.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd8a406c530 0x7fd8a406e9f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:15.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c4151830 0x7fd8c40c6530 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:15.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 --2- 172.21.15.132:0/507657634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8c40b1c90 0x7fd8c40c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:15.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 >> 172.21.15.132:0/507657634 conn(0x7fd8c40cc7a0 msgr2=0x7fd8c406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:15.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 shutdown_connections 2024-03-27T06:15:15.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:15.022+0000 7fd8ca000640 1 -- 172.21.15.132:0/507657634 wait complete. 2024-03-27T06:15:15.032 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:17.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:17.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.257+0000 7fac82800640 1 Processor -- start 2024-03-27T06:15:17.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.257+0000 7fac82800640 1 -- start start 2024-03-27T06:15:17.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.258+0000 7fac82800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fac7c0c9c80 0x7fac7c0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.258+0000 7fac82800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac82800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac7c158670 con 0x7fac7c158290 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac82800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fac7c06ed40 con 0x7fac7c0c9c80 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac82800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fac7c15b2f0 con 0x7fac7c06fb10 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac7be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac72a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fac7c0c9c80 0x7fac7c0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac7be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59472/0 (socket says 172.21.15.132:59472) 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac7b400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fac7c158290 0x7fac7c15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39352/0 (socket says 172.21.15.132:39352) 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac7be00640 1 -- 172.21.15.132:0/3856315806 learned_addr learned my addr 172.21.15.132:0/3856315806 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:17.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac72a00640 1 -- 172.21.15.132:0/3856315806 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 msgr2=0x7fac7c0ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:17.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac72a00640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac72a00640 1 -- 172.21.15.132:0/3856315806 >> v1:172.21.15.132:6789/0 conn(0x7fac7c158290 legacy=0x7fac7c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac72a00640 1 -- 172.21.15.132:0/3856315806 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7c06e2f0 con 0x7fac7c0c9c80 2024-03-27T06:15:17.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.259+0000 7fac7be00640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac72a00640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c0c9c80 0x7fac7c0ce290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac640097a0 tx=0x7fac6402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac114aac966472f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac7aa00640 1 -- 172.21.15.132:0/3856315806 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac64031780 con 0x7fac7c0c9c80 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac7aa00640 1 -- 172.21.15.132:0/3856315806 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac64037070 con 0x7fac7c0c9c80 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac7aa00640 1 -- 172.21.15.132:0/3856315806 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac64031b40 con 0x7fac7c0c9c80 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac82800640 1 -- 172.21.15.132:0/3856315806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c0c9c80 msgr2=0x7fac7c0ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.261+0000 7fac82800640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c0c9c80 0x7fac7c0ce290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac640097a0 tx=0x7fac6402b820 comp rx=0 tx=0).stop 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 -- 172.21.15.132:0/3856315806 shutdown_connections 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c06fb10 0x7fac7c0ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 --2- 172.21.15.132:0/3856315806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c0c9c80 0x7fac7c0ce290 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 -- 172.21.15.132:0/3856315806 >> 172.21.15.132:0/3856315806 conn(0x7fac7c067280 msgr2=0x7fac7c067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 -- 172.21.15.132:0/3856315806 shutdown_connections 2024-03-27T06:15:17.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.262+0000 7fac82800640 1 -- 172.21.15.132:0/3856315806 wait complete. 2024-03-27T06:15:17.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.263+0000 7fac82800640 1 Processor -- start 2024-03-27T06:15:17.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.263+0000 7fac82800640 1 -- start start 2024-03-27T06:15:17.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c0c9c80 0x7fac7c1f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac7c158290 0x7fac7c143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c06f690 con 0x7fac7c06fb10 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c06c9b0 con 0x7fac7c158290 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.264+0000 7fac82800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fac7c06e5d0 con 0x7fac7c0c9c80 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39600/0 (socket says 172.21.15.132:39600) 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 -- 172.21.15.132:0/2267759832 learned_addr learned my addr 172.21.15.132:0/2267759832 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac72a00640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c0c9c80 0x7fac7c1f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7be00640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac7c158290 0x7fac7c143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 -- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c0c9c80 msgr2=0x7fac7c1f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c0c9c80 0x7fac7c1f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 -- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac7c158290 msgr2=0x7fac7c143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac7c158290 0x7fac7c143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 -- 172.21.15.132:0/2267759832 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7c159210 con 0x7fac7c06fb10 2024-03-27T06:15:17.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.265+0000 7fac7b400640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00c730 tx=0x7fac6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:17.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.266+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c017070 con 0x7fac7c06fb10 2024-03-27T06:15:17.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.266+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac6c00ce40 con 0x7fac7c06fb10 2024-03-27T06:15:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.266+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7c1594d0 con 0x7fac7c06fb10 2024-03-27T06:15:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.266+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00a860 con 0x7fac7c06fb10 2024-03-27T06:15:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.266+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac7c0c7d20 con 0x7fac7c06fb10 2024-03-27T06:15:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.267+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac7c068840 con 0x7fac7c06fb10 2024-03-27T06:15:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.268+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fac6c01a070 con 0x7fac7c06fb10 2024-03-27T06:15:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.269+0000 7fac73e00640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 0x7fac5806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.270+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7fac6c08abb0 con 0x7fac7c06fb10 2024-03-27T06:15:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.270+0000 7fac72a00640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 0x7fac5806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.270+0000 7fac72a00640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 0x7fac5806ea40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fac64036000 tx=0x7fac6403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:17.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.274+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fac6c058c50 con 0x7fac7c06fb10 2024-03-27T06:15:17.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.527+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac7c06f8f0 con 0x7fac7c06fb10 2024-03-27T06:15:17.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.529+0000 7fac73e00640 1 -- 172.21.15.132:0/2267759832 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6880 (secure 0 0 0) 0x7fac6c05cf50 con 0x7fac7c06fb10 2024-03-27T06:15:17.520 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:17.521 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4241,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:14:19.619957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/690619582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":690619582},{"type":"v1","addr":"172.21.15.132:6837","nonce":690619582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4244,4270,4273]},"id":1}]} 2024-03-27T06:15:17.521 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 12 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 msgr2=0x7fac5806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 0x7fac5806ea40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fac64036000 tx=0x7fac6403d040 comp rx=0 tx=0).stop 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 msgr2=0x7fac7c0caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00c730 tx=0x7fac6c00cbf0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 shutdown_connections 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fac5806c580 0x7fac5806ea40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac7c158290 0x7fac7c143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fac7c0c9c80 0x7fac7c1f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 --2- 172.21.15.132:0/2267759832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac7c06fb10 0x7fac7c0caaf0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:17.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.531+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 >> 172.21.15.132:0/2267759832 conn(0x7fac7c067280 msgr2=0x7fac7c0c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:17.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.532+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 shutdown_connections 2024-03-27T06:15:17.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:17.532+0000 7fac82800640 1 -- 172.21.15.132:0/2267759832 wait complete. 2024-03-27T06:15:17.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:15:19.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:19.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.760+0000 7fbe5b200640 1 Processor -- start 2024-03-27T06:15:19.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.761+0000 7fbe5b200640 1 -- start start 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5b200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbe5c0cd6a0 0x7fbe5c06b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5b200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fbe5c06be10 0x7fbe5c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5b200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe5c15de10 con 0x7fbe5c1549a0 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5b200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5c0cada0 con 0x7fbe5c06be10 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5b200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5c0cdad0 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe59e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fbe5c1549a0 0x7fbe5c0cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39368/0 (socket says 172.21.15.132:39368) 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe59e00640 1 -- 172.21.15.132:0/814701972 learned_addr learned my addr 172.21.15.132:0/814701972 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe53e00640 1 --2- 172.21.15.132:0/814701972 >> v2:172.21.15.134:3300/0 conn(0x7fbe5c0cd6a0 0x7fbe5c06b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.762+0000 7fbe5a800640 1 --2- 172.21.15.132:0/814701972 >> v2:172.21.15.132:3300/0 conn(0x7fbe5c06be10 0x7fbe5c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.763+0000 7fbe53e00640 1 -- 172.21.15.132:0/814701972 >> v2:172.21.15.132:3300/0 conn(0x7fbe5c06be10 msgr2=0x7fbe5c15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.763+0000 7fbe59400640 1 -- 172.21.15.132:0/814701972 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3202617889 0 0) 0x7fbe5c15de10 con 0x7fbe5c1549a0 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.763+0000 7fbe53e00640 1 --2- 172.21.15.132:0/814701972 >> v2:172.21.15.132:3300/0 conn(0x7fbe5c06be10 0x7fbe5c15adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.763+0000 7fbe53e00640 1 -- 172.21.15.132:0/814701972 >> v1:172.21.15.132:6789/0 conn(0x7fbe5c1549a0 legacy=0x7fbe5c0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.763+0000 7fbe53e00640 1 -- 172.21.15.132:0/814701972 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c15e0b0 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.764+0000 7fbe53e00640 1 --2- 172.21.15.132:0/814701972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c0cd6a0 0x7fbe5c06b8d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0b2df0 tx=0x7fbe4402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d5b720907ca53d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.764+0000 7fbe59400640 1 -- 172.21.15.132:0/814701972 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44034070 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.764+0000 7fbe59400640 1 -- 172.21.15.132:0/814701972 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe44030610 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.764+0000 7fbe59400640 1 -- 172.21.15.132:0/814701972 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440309e0 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 -- 172.21.15.132:0/814701972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c0cd6a0 msgr2=0x7fbe5c06b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 --2- 172.21.15.132:0/814701972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c0cd6a0 0x7fbe5c06b8d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0b2df0 tx=0x7fbe4402b960 comp rx=0 tx=0).stop 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 -- 172.21.15.132:0/814701972 shutdown_connections 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 --2- 172.21.15.132:0/814701972 >> v2:172.21.15.132:3300/0 conn(0x7fbe5c06be10 0x7fbe5c15adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 --2- 172.21.15.132:0/814701972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c0cd6a0 0x7fbe5c06b8d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 -- 172.21.15.132:0/814701972 >> 172.21.15.132:0/814701972 conn(0x7fbe5c06d8d0 msgr2=0x7fbe5c06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 -- 172.21.15.132:0/814701972 shutdown_connections 2024-03-27T06:15:19.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.765+0000 7fbe5b200640 1 -- 172.21.15.132:0/814701972 wait complete. 2024-03-27T06:15:19.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.767+0000 7fbe5b200640 1 Processor -- start 2024-03-27T06:15:19.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.767+0000 7fbe5b200640 1 -- start start 2024-03-27T06:15:19.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbe5c0cd6a0 0x7fbe5c14fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe5c1549a0 0x7fbe5c1469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c15e950 con 0x7fbe5c1549a0 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c15e670 con 0x7fbe5c06be10 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5b200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5c15ec30 con 0x7fbe5c0cd6a0 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:40726/0 (socket says 172.21.15.132:40726) 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe53e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbe5c0cd6a0 0x7fbe5c14fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 -- 172.21.15.132:0/1576962525 learned_addr learned my addr 172.21.15.132:0/1576962525 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe5a800640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe5c1549a0 0x7fbe5c1469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 -- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbe5c0cd6a0 msgr2=0x7fbe5c14fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbe5c0cd6a0 0x7fbe5c14fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 -- 172.21.15.132:0/1576962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe5c1549a0 msgr2=0x7fbe5c1469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:19.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe5c1549a0 0x7fbe5c1469b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.768+0000 7fbe59e00640 1 -- 172.21.15.132:0/1576962525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c150940 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe59e00640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c430 tx=0x7fbe4c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0074e0 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c0047e0 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c004bd0 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5c158060 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.769+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5c14b370 con 0x7fbe5c06be10 2024-03-27T06:15:19.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.770+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe5c1f02e0 con 0x7fbe5c06be10 2024-03-27T06:15:19.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.771+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbe4c011070 con 0x7fbe5c06be10 2024-03-27T06:15:19.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.772+0000 7fbe52000640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 0x7fbe3806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:19.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.772+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbe4c08b230 con 0x7fbe5c06be10 2024-03-27T06:15:19.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.772+0000 7fbe53e00640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 0x7fbe3806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:19.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.773+0000 7fbe53e00640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 0x7fbe3806e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbe44036000 tx=0x7fbe4403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:19.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:19.781+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fbe4c059190 con 0x7fbe5c06be10 2024-03-27T06:15:20.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.028+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe5c0686c0 con 0x7fbe5c06be10 2024-03-27T06:15:20.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.029+0000 7fbe52000640 1 -- 172.21.15.132:0/1576962525 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6052 (secure 0 0 0) 0x7fbe4c05d490 con 0x7fbe5c06be10 2024-03-27T06:15:20.020 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:20.020 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:19.112435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:15:20.020 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 13 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 msgr2=0x7fbe3806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 0x7fbe3806e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbe44036000 tx=0x7fbe4403b040 comp rx=0 tx=0).stop 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 msgr2=0x7fbe5c0c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c430 tx=0x7fbe4c00c8f0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 shutdown_connections 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe5c1549a0 0x7fbe5c1469b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbe5c0cd6a0 0x7fbe5c14fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbe3806c530 0x7fbe3806e9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 --2- 172.21.15.132:0/1576962525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe5c06be10 0x7fbe5c0c7410 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 >> 172.21.15.132:0/1576962525 conn(0x7fbe5c06d8d0 msgr2=0x7fbe5c0cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:20.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 shutdown_connections 2024-03-27T06:15:20.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.032+0000 7fbe5b200640 1 -- 172.21.15.132:0/1576962525 wait complete. 2024-03-27T06:15:20.041 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-03-27T06:15:20.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-27T06:15:20.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:20.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.239+0000 7fd7f2800640 1 Processor -- start 2024-03-27T06:15:20.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.240+0000 7fd7f2800640 1 -- start start 2024-03-27T06:15:20.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7f2800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd7ec152950 0x7fd7ec1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7f2800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd7ec073970 0x7fd7ec06cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7f2800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec06b630 con 0x7fd7ec073550 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7f2800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec152d30 con 0x7fd7ec152950 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7f2800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec06b8d0 con 0x7fd7ec073970 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7eb400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd7ec152950 0x7fd7ec1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7ebe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd7ec073970 0x7fd7ec06cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7eaa00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd7ec073550 0x7fd7ec071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:52912/0 (socket says 172.21.15.132:52912) 2024-03-27T06:15:20.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.241+0000 7fd7eaa00640 1 -- 172.21.15.132:0/496496907 learned_addr learned my addr 172.21.15.132:0/496496907 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.242+0000 7fd7ea000640 1 -- 172.21.15.132:0/496496907 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832457318 0 0) 0x7fd7ec06b630 con 0x7fd7ec073550 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.242+0000 7fd7ebe00640 1 -- 172.21.15.132:0/496496907 >> v2:172.21.15.132:3300/0 conn(0x7fd7ec152950 msgr2=0x7fd7ec1590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.242+0000 7fd7ebe00640 1 --2- 172.21.15.132:0/496496907 >> v2:172.21.15.132:3300/0 conn(0x7fd7ec152950 0x7fd7ec1590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.242+0000 7fd7ebe00640 1 -- 172.21.15.132:0/496496907 >> v1:172.21.15.134:6789/0 conn(0x7fd7ec073550 legacy=0x7fd7ec071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.242+0000 7fd7ebe00640 1 -- 172.21.15.132:0/496496907 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec06bbb0 con 0x7fd7ec073970 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.243+0000 7fd7ebe00640 1 --2- 172.21.15.132:0/496496907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec073970 0x7fd7ec06cf50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c928785fdb5802a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.243+0000 7fd7ea000640 1 -- 172.21.15.132:0/496496907 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4039070 con 0x7fd7ec073970 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.243+0000 7fd7ea000640 1 -- 172.21.15.132:0/496496907 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e402cc60 con 0x7fd7ec073970 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.243+0000 7fd7ea000640 1 -- 172.21.15.132:0/496496907 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4034660 con 0x7fd7ec073970 2024-03-27T06:15:20.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 -- 172.21.15.132:0/496496907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec073970 msgr2=0x7fd7ec06cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 --2- 172.21.15.132:0/496496907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec073970 0x7fd7ec06cf50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009d90 tx=0x7fd7e402beb0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 -- 172.21.15.132:0/496496907 shutdown_connections 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 --2- 172.21.15.132:0/496496907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec073970 0x7fd7ec06cf50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 --2- 172.21.15.132:0/496496907 >> v2:172.21.15.132:3300/0 conn(0x7fd7ec152950 0x7fd7ec1590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 -- 172.21.15.132:0/496496907 >> 172.21.15.132:0/496496907 conn(0x7fd7ec0cc060 msgr2=0x7fd7ec0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 -- 172.21.15.132:0/496496907 shutdown_connections 2024-03-27T06:15:20.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.244+0000 7fd7f2800640 1 -- 172.21.15.132:0/496496907 wait complete. 2024-03-27T06:15:20.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.245+0000 7fd7f2800640 1 Processor -- start 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.245+0000 7fd7f2800640 1 -- start start 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 0x7fd7ec1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7ec073970 0x7fd7ec157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec06c450 con 0x7fd7ec073970 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec06c170 con 0x7fd7ec152950 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.246+0000 7fd7f2800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec071880 con 0x7fd7ec073550 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7ec073970 0x7fd7ec157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7eb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 0x7fd7ec1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7eb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 0x7fd7ec1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59488/0 (socket says 172.21.15.132:59488) 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:40744/0 (socket says 172.21.15.132:40744) 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7eb400640 1 -- 172.21.15.132:0/1774936523 learned_addr learned my addr 172.21.15.132:0/1774936523 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 -- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 msgr2=0x7fd7ec1f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 0x7fd7ec1f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 -- 172.21.15.132:0/1774936523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7ec073970 msgr2=0x7fd7ec157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7ec073970 0x7fd7ec157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 -- 172.21.15.132:0/1774936523 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec153490 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.247+0000 7fd7ebe00640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4033000 tx=0x7fd7e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.248+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4039050 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.248+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e4045070 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.248+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec153770 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.248+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4009450 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.248+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec1f0500 con 0x7fd7ec152950 2024-03-27T06:15:20.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.249+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec068840 con 0x7fd7ec152950 2024-03-27T06:15:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.250+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd7e403e850 con 0x7fd7ec152950 2024-03-27T06:15:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.250+0000 7fd7e3400640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.251+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd7e40ae480 con 0x7fd7ec152950 2024-03-27T06:15:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.251+0000 7fd7eb400640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.251+0000 7fd7eb400640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009b60 tx=0x7fd7dc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:20.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.259+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd7e407c3f0 con 0x7fd7ec152950 2024-03-27T06:15:20.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.510+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7ec0c6b30 con 0x7fd7ec152950 2024-03-27T06:15:20.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.510+0000 7fd7e3400640 1 -- 172.21.15.132:0/1774936523 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6052 (secure 0 0 0) 0x7fd7e40806f0 con 0x7fd7ec152950 2024-03-27T06:15:20.501 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:20.501 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:19.112435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:15:20.502 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 13 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 msgr2=0x7fd7cc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009b60 tx=0x7fd7dc009190 comp rx=0 tx=0).stop 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 msgr2=0x7fd7ec1f4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4033000 tx=0x7fd7e4004a40 comp rx=0 tx=0).stop 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 shutdown_connections 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd7cc06c530 0x7fd7cc06e9f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7ec152950 0x7fd7ec1f4090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7ec073970 0x7fd7ec157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 --2- 172.21.15.132:0/1774936523 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd7ec073550 0x7fd7ec1f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:20.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 >> 172.21.15.132:0/1774936523 conn(0x7fd7ec0cc060 msgr2=0x7fd7ec071170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:20.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 shutdown_connections 2024-03-27T06:15:20.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:20.513+0000 7fd7f2800640 1 -- 172.21.15.132:0/1774936523 wait complete. 2024-03-27T06:15:20.525 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4241": { "addr": "172.21.15.132:6836/3973814804", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 3973814804, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 3973814804, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 13, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4270": { "addr": "172.21.15.134:6838/2702381292", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2702381292, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2702381292, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:15:19.112435+0000", "qdb_cluster": [ 4270, 4273 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4270, "mds_2": 4241 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:15:22.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:22.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.737+0000 7feeb2400640 1 Processor -- start 2024-03-27T06:15:22.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.737+0000 7feeb2400640 1 -- start start 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feeb2400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7feeac1549a0 0x7feeac0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feeb2400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeac15de10 con 0x7feeac15adb0 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feeb2400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeac15e0b0 con 0x7feeac0cb870 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feeb2400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7feeac0cada0 con 0x7feeac1549a0 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feeb1000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7feeac1549a0 0x7feeac0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feea3e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7feeac0cb870 0x7feeac06b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:59008/0 (socket says 172.21.15.132:59008) 2024-03-27T06:15:22.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.739+0000 7feea3e00640 1 -- 172.21.15.132:0/188541283 learned_addr learned my addr 172.21.15.132:0/188541283 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590436411 0 0) 0x7feeac15e0b0 con 0x7feeac0cb870 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee90003520 con 0x7feeac0cb870 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847645843 0 0) 0x7feeac15de10 con 0x7feeac15adb0 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeb1000640 1 -- 172.21.15.132:0/188541283 >> v1:172.21.15.134:6789/0 conn(0x7feeac0cb870 legacy=0x7feeac06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeb1000640 1 -- 172.21.15.132:0/188541283 >> v1:172.21.15.132:6789/0 conn(0x7feeac15adb0 legacy=0x7feeac15b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:22.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.740+0000 7feeb1000640 1 -- 172.21.15.132:0/188541283 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac15e390 con 0x7feeac1549a0 2024-03-27T06:15:22.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.742+0000 7feeb1000640 1 --2- 172.21.15.132:0/188541283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac1549a0 0x7feeac0cd6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0097a0 tx=0x7fee9c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc15b34bf92fc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:22.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.742+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c02f810 con 0x7feeac1549a0 2024-03-27T06:15:22.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.742+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9c02f9b0 con 0x7feeac1549a0 2024-03-27T06:15:22.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.742+0000 7feeabe00640 1 -- 172.21.15.132:0/188541283 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c02fd60 con 0x7feeac1549a0 2024-03-27T06:15:22.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 -- 172.21.15.132:0/188541283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac1549a0 msgr2=0x7feeac0cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:22.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 --2- 172.21.15.132:0/188541283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac1549a0 0x7feeac0cd6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0097a0 tx=0x7fee9c02b9f0 comp rx=0 tx=0).stop 2024-03-27T06:15:22.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 -- 172.21.15.132:0/188541283 shutdown_connections 2024-03-27T06:15:22.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 --2- 172.21.15.132:0/188541283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac1549a0 0x7feeac0cd6a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:22.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 -- 172.21.15.132:0/188541283 >> 172.21.15.132:0/188541283 conn(0x7feeac06d8d0 msgr2=0x7feeac06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:22.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 -- 172.21.15.132:0/188541283 shutdown_connections 2024-03-27T06:15:22.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.743+0000 7feeb2400640 1 -- 172.21.15.132:0/188541283 wait complete. 2024-03-27T06:15:22.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.744+0000 7feeb2400640 1 Processor -- start 2024-03-27T06:15:22.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.745+0000 7feeb2400640 1 -- start start 2024-03-27T06:15:22.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.745+0000 7feeb2400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7feeac0cb870 0x7feeac0b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.745+0000 7feeb2400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb2400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac15adb0 0x7feeac146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb2400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feeac15ec30 con 0x7feeac15adb0 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb2400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7feeac15e950 con 0x7feeac1549a0 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb2400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7feeac15a820 con 0x7feeac0cb870 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57210/0 (socket says 172.21.15.132:57210) 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 -- 172.21.15.132:0/2336101534 learned_addr learned my addr 172.21.15.132:0/2336101534 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb1000640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7feeac0cb870 0x7feeac0b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feeb1a00640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac15adb0 0x7feeac146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 -- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7feeac0cb870 msgr2=0x7feeac0b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7feeac0cb870 0x7feeac0b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 -- 172.21.15.132:0/2336101534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac15adb0 msgr2=0x7feeac146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:22.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac15adb0 0x7feeac146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:22.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 -- 172.21.15.132:0/2336101534 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac1523a0 con 0x7feeac1549a0 2024-03-27T06:15:22.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.746+0000 7feea3e00640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c390 tx=0x7fee9400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:22.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.747+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94017070 con 0x7feeac1549a0 2024-03-27T06:15:22.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.747+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee94005ce0 con 0x7feeac1549a0 2024-03-27T06:15:22.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.747+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940072d0 con 0x7feeac1549a0 2024-03-27T06:15:22.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.747+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac14e6d0 con 0x7feeac1549a0 2024-03-27T06:15:22.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.747+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac0b43b0 con 0x7feeac1549a0 2024-03-27T06:15:22.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.748+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeac0cc650 con 0x7feeac1549a0 2024-03-27T06:15:22.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.749+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fee9401a070 con 0x7feeac1549a0 2024-03-27T06:15:22.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.750+0000 7feea9600640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 0x7fee8806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:22.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.750+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fee9408a950 con 0x7feeac1549a0 2024-03-27T06:15:22.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.750+0000 7feeb1000640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 0x7fee8806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:22.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.751+0000 7feeb1000640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 0x7fee8806e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fee9c036000 tx=0x7fee9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:22.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:22.759+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7feeac0cc650 con 0x7feeac1549a0 2024-03-27T06:15:23.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.028+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feeac0b45b0 con 0x7feeac1549a0 2024-03-27T06:15:23.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.029+0000 7feea9600640 1 -- 172.21.15.132:0/2336101534 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6052 (secure 0 0 0) 0x7feeac0b45b0 con 0x7feeac1549a0 2024-03-27T06:15:23.019 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:23.020 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:19.112435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:15:23.020 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 13 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 msgr2=0x7fee8806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 0x7fee8806e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fee9c036000 tx=0x7fee9c008030 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 msgr2=0x7feeac1469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c390 tx=0x7fee9400c850 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 shutdown_connections 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feeac15adb0 0x7feeac146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fee8806c530 0x7fee8806e9f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feeac1549a0 0x7feeac1469b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 --2- 172.21.15.132:0/2336101534 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7feeac0cb870 0x7feeac0b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 >> 172.21.15.132:0/2336101534 conn(0x7feeac06d8d0 msgr2=0x7feeac0cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 shutdown_connections 2024-03-27T06:15:23.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:23.031+0000 7feeb2400640 1 -- 172.21.15.132:0/2336101534 wait complete. 2024-03-27T06:15:23.041 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:25.042 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:25.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.248+0000 7f1992000640 1 Processor -- start 2024-03-27T06:15:25.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.248+0000 7f1992000640 1 -- start start 2024-03-27T06:15:25.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.250+0000 7f1992000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f198c0b11e0 0x7f198c0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:25.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.250+0000 7f1992000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c15b790 con 0x7f198c15aa00 2024-03-27T06:15:25.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f1992000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c0c12d0 con 0x7f198c0c6fa0 2024-03-27T06:15:25.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f1992000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f198c0c1080 con 0x7f198c0b11e0 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f1983e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f198c0c6fa0 0x7f198c06b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:59020/0 (socket says 172.21.15.132:59020) 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f198c0b11e0 0x7f198c0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f1983e00640 1 -- 172.21.15.132:0/3685987493 learned_addr learned my addr 172.21.15.132:0/3685987493 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198aa00640 1 -- 172.21.15.132:0/3685987493 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484707545 0 0) 0x7f198c0c12d0 con 0x7f198c0c6fa0 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198aa00640 1 -- 172.21.15.132:0/3685987493 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1970003520 con 0x7f198c0c6fa0 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198b400640 1 -- 172.21.15.132:0/3685987493 >> v1:172.21.15.134:6789/0 conn(0x7f198c0c6fa0 legacy=0x7f198c06b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198b400640 1 -- 172.21.15.132:0/3685987493 >> v1:172.21.15.132:6789/0 conn(0x7f198c15aa00 legacy=0x7f198c15ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.251+0000 7f198b400640 1 -- 172.21.15.132:0/3685987493 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c0c15b0 con 0x7f198c0b11e0 2024-03-27T06:15:25.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.252+0000 7f198b400640 1 --2- 172.21.15.132:0/3685987493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c0c6a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f197c0097a0 tx=0x7f197c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7342021f982c7a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:25.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f198aa00640 1 -- 172.21.15.132:0/3685987493 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c02f810 con 0x7f198c0b11e0 2024-03-27T06:15:25.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f198aa00640 1 -- 172.21.15.132:0/3685987493 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c02f9b0 con 0x7f198c0b11e0 2024-03-27T06:15:25.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f198aa00640 1 -- 172.21.15.132:0/3685987493 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c02fd80 con 0x7f198c0b11e0 2024-03-27T06:15:25.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 -- 172.21.15.132:0/3685987493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 msgr2=0x7f198c0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 --2- 172.21.15.132:0/3685987493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c0c6a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f197c0097a0 tx=0x7f197c02b9f0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 -- 172.21.15.132:0/3685987493 shutdown_connections 2024-03-27T06:15:25.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 --2- 172.21.15.132:0/3685987493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c0c6a60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 -- 172.21.15.132:0/3685987493 >> 172.21.15.132:0/3685987493 conn(0x7f198c0cc060 msgr2=0x7f198c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:25.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.253+0000 7f1992000640 1 -- 172.21.15.132:0/3685987493 shutdown_connections 2024-03-27T06:15:25.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.254+0000 7f1992000640 1 -- 172.21.15.132:0/3685987493 wait complete. 2024-03-27T06:15:25.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.256+0000 7f1992000640 1 Processor -- start 2024-03-27T06:15:25.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.256+0000 7f1992000640 1 -- start start 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198c0c6fa0 0x7f198c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198c15aa00 0x7f198c1f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f198c0b15c0 con 0x7f198c0b11e0 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f198c0c1c00 con 0x7f198c0c6fa0 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1992000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f198c0c2090 con 0x7f198c15aa00 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38114/0 (socket says 172.21.15.132:38114) 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 -- 172.21.15.132:0/4173568411 learned_addr learned my addr 172.21.15.132:0/4173568411 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:25.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f1983e00640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198c0c6fa0 0x7f198c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198be00640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198c15aa00 0x7f198c1f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 -- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198c15aa00 msgr2=0x7f198c1f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198c15aa00 0x7f198c1f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 -- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198c0c6fa0 msgr2=0x7f198c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198c0c6fa0 0x7f198c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.258+0000 7f198b400640 1 -- 172.21.15.132:0/4173568411 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c1e6320 con 0x7f198c0b11e0 2024-03-27T06:15:25.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f198b400640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f197c02c000 tx=0x7f197c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c047070 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c0378f0 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c037ce0 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c155200 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.259+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c0c37d0 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.260+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198c068840 con 0x7f198c0b11e0 2024-03-27T06:15:25.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.261+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f197c04a070 con 0x7f198c0b11e0 2024-03-27T06:15:25.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.263+0000 7f1983400640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:25.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.263+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f197c0ad250 con 0x7f198c0b11e0 2024-03-27T06:15:25.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.263+0000 7f1983e00640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:25.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.264+0000 7f1983e00640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f198c0cb220 tx=0x7f1974007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:25.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.268+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f197c033030 con 0x7f198c0b11e0 2024-03-27T06:15:25.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.544+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f198c158360 con 0x7f198c0b11e0 2024-03-27T06:15:25.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.545+0000 7f1983400640 1 -- 172.21.15.132:0/4173568411 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6054 (secure 0 0 0) 0x7f197c07b0f0 con 0x7f198c0b11e0 2024-03-27T06:15:25.536 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:25.536 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:24.131101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:resolve","state_seq":24,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:15:25.536 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 14 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 msgr2=0x7f196806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f198c0cb220 tx=0x7f1974007d30 comp rx=0 tx=0).stop 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 msgr2=0x7f198c154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f197c02c000 tx=0x7f197c004730 comp rx=0 tx=0).stop 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 shutdown_connections 2024-03-27T06:15:25.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198c15aa00 0x7f198c1f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198c0c6fa0 0x7f198c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 --2- 172.21.15.132:0/4173568411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198c0b11e0 0x7f198c154cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 >> 172.21.15.132:0/4173568411 conn(0x7f198c0cc060 msgr2=0x7f198c071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.547+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 shutdown_connections 2024-03-27T06:15:25.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:25.548+0000 7f1992000640 1 -- 172.21.15.132:0/4173568411 wait complete. 2024-03-27T06:15:25.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:27.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:27.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.764+0000 7f4de3a00640 1 Processor -- start 2024-03-27T06:15:27.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.764+0000 7f4de3a00640 1 -- start start 2024-03-27T06:15:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4ddc154e00 0x7f4ddc15b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4ddc15ba80 con 0x7f4ddc154e00 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f4ddc06ba60 con 0x7f4ddc15bf30 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de3a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4ddc06bd40 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4ddbe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4ddc154e00 0x7f4ddc15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4ddbe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57230/0 (socket says 172.21.15.132:57230) 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:48794/0 (socket says 172.21.15.132:48794) 2024-03-27T06:15:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4ddbe00640 1 -- 172.21.15.132:0/1966406576 learned_addr learned my addr 172.21.15.132:0/1966406576 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 -- 172.21.15.132:0/1966406576 >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 msgr2=0x7f4ddc0c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 --2- 172.21.15.132:0/1966406576 >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 -- 172.21.15.132:0/1966406576 >> v2:172.21.15.132:3300/0 conn(0x7f4ddc154e00 msgr2=0x7f4ddc15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 --2- 172.21.15.132:0/1966406576 >> v2:172.21.15.132:3300/0 conn(0x7f4ddc154e00 0x7f4ddc15b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.766+0000 7f4de1a00640 1 -- 172.21.15.132:0/1966406576 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ddc0c8230 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.767+0000 7f4ddbe00640 1 --2- 172.21.15.132:0/1966406576 >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.767+0000 7f4de1a00640 1 --2- 172.21.15.132:0/1966406576 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc08d6e0 tx=0x7f4dd402be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=144b2f1156d1e607 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4ddb400640 1 -- 172.21.15.132:0/1966406576 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4039070 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4ddb400640 1 -- 172.21.15.132:0/1966406576 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd402cbe0 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4ddb400640 1 -- 172.21.15.132:0/1966406576 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4034440 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 -- 172.21.15.132:0/1966406576 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 msgr2=0x7f4ddc0c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 --2- 172.21.15.132:0/1966406576 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc08d6e0 tx=0x7f4dd402be30 comp rx=0 tx=0).stop 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 -- 172.21.15.132:0/1966406576 shutdown_connections 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 --2- 172.21.15.132:0/1966406576 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc06c3d0 0x7f4ddc0c7550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 --2- 172.21.15.132:0/1966406576 >> v2:172.21.15.134:3300/0 conn(0x7f4ddc15bf30 0x7f4ddc0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 --2- 172.21.15.132:0/1966406576 >> v2:172.21.15.132:3300/0 conn(0x7f4ddc154e00 0x7f4ddc15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 -- 172.21.15.132:0/1966406576 >> 172.21.15.132:0/1966406576 conn(0x7f4ddc0d0150 msgr2=0x7f4ddc0d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.768+0000 7f4de3a00640 1 -- 172.21.15.132:0/1966406576 shutdown_connections 2024-03-27T06:15:27.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.769+0000 7f4de3a00640 1 -- 172.21.15.132:0/1966406576 wait complete. 2024-03-27T06:15:27.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.770+0000 7f4de3a00640 1 Processor -- start 2024-03-27T06:15:27.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.770+0000 7f4de3a00640 1 -- start start 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 0x7f4ddc157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc1551e0 con 0x7f4ddc06c3d0 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc0c87f0 con 0x7f4ddc154e00 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de3a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4ddc0c9250 con 0x7f4ddc15bf30 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4ddbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de1a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de1000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 0x7f4ddc157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4ddbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57236/0 (socket says 172.21.15.132:57236) 2024-03-27T06:15:27.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4ddbe00640 1 -- 172.21.15.132:0/66436959 learned_addr learned my addr 172.21.15.132:0/66436959 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:27.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.771+0000 7f4de1a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:48798/0 (socket says 172.21.15.132:48798) 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 -- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 msgr2=0x7f4ddc0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 -- 172.21.15.132:0/66436959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 msgr2=0x7f4ddc157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 0x7f4ddc157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 -- 172.21.15.132:0/66436959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ddc1f4da0 con 0x7f4ddc154e00 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4de1a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4de1000640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 0x7f4ddc157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4ddbe00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e980 tx=0x7f4dc800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8017070 con 0x7f4ddc154e00 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc800ca00 con 0x7f4ddc154e00 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ddc1f4ac0 con 0x7f4ddc154e00 2024-03-27T06:15:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.772+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800cdf0 con 0x7f4ddc154e00 2024-03-27T06:15:27.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.773+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ddc0ce340 con 0x7f4ddc154e00 2024-03-27T06:15:27.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.774+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ddc0c9e00 con 0x7f4ddc154e00 2024-03-27T06:15:27.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.775+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4dc801a070 con 0x7f4ddc154e00 2024-03-27T06:15:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.775+0000 7f4dd8c00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 0x7f4dbc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.776+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4dc808b4c0 con 0x7f4ddc154e00 2024-03-27T06:15:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.776+0000 7f4de1000640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 0x7f4dbc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.776+0000 7f4de1000640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 0x7f4dbc06ea20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009e90 tx=0x7f4dd0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:27.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:27.780+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4ddc0c9e00 con 0x7f4ddc154e00 2024-03-27T06:15:28.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.014+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ddc068840 con 0x7f4ddc154e00 2024-03-27T06:15:28.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.014+0000 7f4dd8c00640 1 -- 172.21.15.132:0/66436959 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6054 (secure 0 0 0) 0x7f4dc8059470 con 0x7f4ddc154e00 2024-03-27T06:15:28.005 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:28.005 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:24.131101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:resolve","state_seq":24,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:15:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 14 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 msgr2=0x7f4dbc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 0x7f4dbc06ea20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009e90 tx=0x7f4dd0009190 comp rx=0 tx=0).stop 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 msgr2=0x7f4ddc1f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e980 tx=0x7f4dc800ee40 comp rx=0 tx=0).stop 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 shutdown_connections 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4ddc15bf30 0x7f4ddc0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4dbc06c560 0x7f4dbc06ea20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ddc154e00 0x7f4ddc1f11c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 --2- 172.21.15.132:0/66436959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ddc06c3d0 0x7f4ddc157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:28.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 >> 172.21.15.132:0/66436959 conn(0x7f4ddc0d0150 msgr2=0x7f4ddc14fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:28.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 shutdown_connections 2024-03-27T06:15:28.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:28.017+0000 7f4de3a00640 1 -- 172.21.15.132:0/66436959 wait complete. 2024-03-27T06:15:28.024 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:30.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:30.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.259+0000 7fe793200640 1 Processor -- start 2024-03-27T06:15:30.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.259+0000 7fe793200640 1 -- start start 2024-03-27T06:15:30.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.260+0000 7fe793200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe7941549a0 0x7fe7940cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.260+0000 7fe793200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe7940cb870 0x7fe79406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe793200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79415de10 con 0x7fe79406be10 2024-03-27T06:15:30.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe793200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fe7940cada0 con 0x7fe7940cb870 2024-03-27T06:15:30.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe793200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fe79415e0b0 con 0x7fe7941549a0 2024-03-27T06:15:30.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe7941549a0 0x7fe7940cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe7941549a0 0x7fe7940cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57242/0 (socket says 172.21.15.132:57242) 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 -- 172.21.15.132:0/2401998643 learned_addr learned my addr 172.21.15.132:0/2401998643 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791400640 1 --2- 172.21.15.132:0/2401998643 >> v2:172.21.15.132:3300/0 conn(0x7fe7940cb870 0x7fe79406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 -- 172.21.15.132:0/2401998643 >> v2:172.21.15.132:3300/0 conn(0x7fe7940cb870 msgr2=0x7fe79406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 --2- 172.21.15.132:0/2401998643 >> v2:172.21.15.132:3300/0 conn(0x7fe7940cb870 0x7fe79406b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe791e00640 1 -- 172.21.15.132:0/2401998643 >> v1:172.21.15.134:6789/0 conn(0x7fe79406be10 legacy=0x7fe79415adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.261+0000 7fe790a00640 1 -- 172.21.15.132:0/2401998643 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702983621 0 0) 0x7fe79415de10 con 0x7fe79406be10 2024-03-27T06:15:30.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.262+0000 7fe791e00640 1 -- 172.21.15.132:0/2401998643 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79415e390 con 0x7fe7941549a0 2024-03-27T06:15:30.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.263+0000 7fe791e00640 1 --2- 172.21.15.132:0/2401998643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7941549a0 0x7fe7940cd6a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7940c73e0 tx=0x7fe78402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca2b7e2b3225d94b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.263+0000 7fe790a00640 1 -- 172.21.15.132:0/2401998643 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78402cbb0 con 0x7fe7941549a0 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.263+0000 7fe790a00640 1 -- 172.21.15.132:0/2401998643 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78402cd50 con 0x7fe7941549a0 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.263+0000 7fe790a00640 1 -- 172.21.15.132:0/2401998643 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784034610 con 0x7fe7941549a0 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 -- 172.21.15.132:0/2401998643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7941549a0 msgr2=0x7fe7940cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 --2- 172.21.15.132:0/2401998643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7941549a0 0x7fe7940cd6a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7940c73e0 tx=0x7fe78402c010 comp rx=0 tx=0).stop 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 -- 172.21.15.132:0/2401998643 shutdown_connections 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 --2- 172.21.15.132:0/2401998643 >> v2:172.21.15.132:3300/0 conn(0x7fe7940cb870 0x7fe79406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 --2- 172.21.15.132:0/2401998643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7941549a0 0x7fe7940cd6a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 -- 172.21.15.132:0/2401998643 >> 172.21.15.132:0/2401998643 conn(0x7fe79406d8d0 msgr2=0x7fe79406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 -- 172.21.15.132:0/2401998643 shutdown_connections 2024-03-27T06:15:30.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.264+0000 7fe793200640 1 -- 172.21.15.132:0/2401998643 wait complete. 2024-03-27T06:15:30.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.266+0000 7fe793200640 1 Processor -- start 2024-03-27T06:15:30.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.267+0000 7fe793200640 1 -- start start 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 0x7fe7940b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 0x7fe794147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe79415ec30 con 0x7fe79406be10 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe79415e950 con 0x7fe7940cb870 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe793200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe79415a820 con 0x7fe7941549a0 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe791e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe791e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38138/0 (socket says 172.21.15.132:38138) 2024-03-27T06:15:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe791e00640 1 -- 172.21.15.132:0/2672305136 learned_addr learned my addr 172.21.15.132:0/2672305136 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.268+0000 7fe792800640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 0x7fe794147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe791400640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 0x7fe7940b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 -- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 msgr2=0x7fe7940b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 0x7fe7940b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 -- 172.21.15.132:0/2672305136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 msgr2=0x7fe794149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 -- 172.21.15.132:0/2672305136 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79415b990 con 0x7fe7941549a0 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe791e00640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe791400640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 0x7fe7940b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.269+0000 7fe792800640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 0x7fe794147730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c3e0 tx=0x7fe78c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.270+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c017070 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.270+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c0035f0 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.270+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe794150c10 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.270+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7941f0450 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.270+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0039e0 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.271+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76c005290 con 0x7fe7941549a0 2024-03-27T06:15:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.272+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe78c01a070 con 0x7fe7941549a0 2024-03-27T06:15:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.273+0000 7fe78a000640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 0x7fe77406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.273+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe78c08aa80 con 0x7fe7941549a0 2024-03-27T06:15:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.274+0000 7fe791e00640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 0x7fe77406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.274+0000 7fe791e00640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 0x7fe77406e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe784000f80 tx=0x7fe784030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.277+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe78c08ce20 con 0x7fe7941549a0 2024-03-27T06:15:30.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.717+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe76c005580 con 0x7fe7941549a0 2024-03-27T06:15:30.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.717+0000 7fe78a000640 1 -- 172.21.15.132:0/2672305136 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6056 (secure 0 0 0) 0x7fe78c058960 con 0x7fe7941549a0 2024-03-27T06:15:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:30.708 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:29.846590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:15:30.708 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 15 2024-03-27T06:15:30.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.719+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 msgr2=0x7fe77406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.719+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 0x7fe77406e9f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe784000f80 tx=0x7fe784030000 comp rx=0 tx=0).stop 2024-03-27T06:15:30.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.719+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 msgr2=0x7fe794147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:30.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.719+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 0x7fe794147730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c3e0 tx=0x7fe78c00c8a0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 shutdown_connections 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe7941549a0 0x7fe794147730 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe77406c530 0x7fe77406e9f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7940cb870 0x7fe7940b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 --2- 172.21.15.132:0/2672305136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe79406be10 0x7fe794149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 >> 172.21.15.132:0/2672305136 conn(0x7fe79406d8d0 msgr2=0x7fe7940d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 shutdown_connections 2024-03-27T06:15:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:30.720+0000 7fe793200640 1 -- 172.21.15.132:0/2672305136 wait complete. 2024-03-27T06:15:30.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:32.734 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:32.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.937+0000 7f1987200640 1 Processor -- start 2024-03-27T06:15:32.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.937+0000 7f1987200640 1 -- start start 2024-03-27T06:15:32.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1987200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f198806ce00 0x7f19880cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1987200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 0x7f198815b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1987200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198815ebe0 con 0x7f19881548b0 2024-03-27T06:15:32.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1987200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f198815b6a0 con 0x7f198806ce00 2024-03-27T06:15:32.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1987200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f19880cbc80 con 0x7f198815ad60 2024-03-27T06:15:32.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1985400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f198806ce00 0x7f19880cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1985400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f198806ce00 0x7f19880cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57318/0 (socket says 172.21.15.132:57318) 2024-03-27T06:15:32.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1985400640 1 -- 172.21.15.132:0/1350630428 learned_addr learned my addr 172.21.15.132:0/1350630428 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1986800640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 0x7f198815b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.939+0000 7f1985400640 1 -- 172.21.15.132:0/1350630428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 msgr2=0x7f198815b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.940+0000 7f1985400640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 0x7f198815b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.940+0000 7f1985400640 1 -- 172.21.15.132:0/1350630428 >> v1:172.21.15.132:6789/0 conn(0x7f19881548b0 legacy=0x7f19880cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.940+0000 7f1985400640 1 -- 172.21.15.132:0/1350630428 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198815e060 con 0x7f198806ce00 2024-03-27T06:15:32.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.940+0000 7f1986800640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 0x7f198815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:32.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.941+0000 7f1985400640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198806ce00 0x7f19880cb740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1974009870 tx=0x7f197402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f013a4b7731da8e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:32.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.941+0000 7f1984a00640 1 -- 172.21.15.132:0/1350630428 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197402f830 con 0x7f198806ce00 2024-03-27T06:15:32.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.941+0000 7f1984a00640 1 -- 172.21.15.132:0/1350630428 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197402f9d0 con 0x7f198806ce00 2024-03-27T06:15:32.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.941+0000 7f1984a00640 1 -- 172.21.15.132:0/1350630428 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197402fda0 con 0x7f198806ce00 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 -- 172.21.15.132:0/1350630428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198806ce00 msgr2=0x7f19880cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198806ce00 0x7f19880cb740 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1974009870 tx=0x7f197402ba10 comp rx=0 tx=0).stop 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 -- 172.21.15.132:0/1350630428 shutdown_connections 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f198815ad60 0x7f198815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 --2- 172.21.15.132:0/1350630428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198806ce00 0x7f19880cb740 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 -- 172.21.15.132:0/1350630428 >> 172.21.15.132:0/1350630428 conn(0x7f198806e190 msgr2=0x7f1988067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 -- 172.21.15.132:0/1350630428 shutdown_connections 2024-03-27T06:15:32.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.942+0000 7f1987200640 1 -- 172.21.15.132:0/1350630428 wait complete. 2024-03-27T06:15:32.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.944+0000 7f1987200640 1 Processor -- start 2024-03-27T06:15:32.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.944+0000 7f1987200640 1 -- start start 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 0x7f19881f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 0x7f19880cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f198815e900 con 0x7f198815ad60 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f198815e620 con 0x7f198806ce00 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1987200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f198815a840 con 0x7f19881548b0 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1985e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1985e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:49334/0 (socket says 172.21.15.132:49334) 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1985e00640 1 -- 172.21.15.132:0/2489582969 learned_addr learned my addr 172.21.15.132:0/2489582969 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1985400640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 0x7f19881f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.945+0000 7f1986800640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 0x7f19880cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 -- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 msgr2=0x7f19881f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:32.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 0x7f19881f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 -- 172.21.15.132:0/2489582969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 msgr2=0x7f19880cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 0x7f19880cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 -- 172.21.15.132:0/2489582969 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19880b4e10 con 0x7f198806ce00 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1986800640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 0x7f19880cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985400640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 0x7f19881f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1985e00640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f197800c430 tx=0x7f197800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780052d0 con 0x7f198806ce00 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1978005490 con 0x7f198806ce00 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978007260 con 0x7f198806ce00 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198815b900 con 0x7f198806ce00 2024-03-27T06:15:32.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.946+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198815bec0 con 0x7f198806ce00 2024-03-27T06:15:32.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.948+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1988154460 con 0x7f198806ce00 2024-03-27T06:15:32.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.949+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1978007400 con 0x7f198806ce00 2024-03-27T06:15:32.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.950+0000 7f197e000640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:32.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.950+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f197808abb0 con 0x7f198806ce00 2024-03-27T06:15:32.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.950+0000 7f1985400640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:32.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.950+0000 7f1985400640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f1974008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:32.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:32.953+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f1988154460 con 0x7f198806ce00 2024-03-27T06:15:33.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:15:33.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.244+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19880686c0 con 0x7f198806ce00 2024-03-27T06:15:33.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:15:33.239 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:33.239 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:29.846590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.245+0000 7f197e000640 1 -- 172.21.15.132:0/2489582969 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6056 (secure 0 0 0) 0x7f19880686c0 con 0x7f198806ce00 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 15 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 msgr2=0x7f196806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f1974008030 comp rx=0 tx=0).stop 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 msgr2=0x7f198814a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f197800c430 tx=0x7f197800c8f0 comp rx=0 tx=0).stop 2024-03-27T06:15:33.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 shutdown_connections 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f198815ad60 0x7f19880cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f19881548b0 0x7f19881f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f196806c530 0x7f196806e9f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 --2- 172.21.15.132:0/2489582969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f198806ce00 0x7f198814a3a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 >> 172.21.15.132:0/2489582969 conn(0x7f198806e190 msgr2=0x7f19880cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 shutdown_connections 2024-03-27T06:15:33.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:33.249+0000 7f1987200640 1 -- 172.21.15.132:0/2489582969 wait complete. 2024-03-27T06:15:33.260 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4241": { "addr": "172.21.15.132:6836/3973814804", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 3973814804, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 3973814804, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4241, "incarnation": 13, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 26 }, "gid_4270": { "addr": "172.21.15.134:6838/2702381292", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2702381292, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2702381292, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:15:29.846590+0000", "qdb_cluster": [ 4270, 4273 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4270, "mds_2": 4241 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:15:35.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:35.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.444+0000 7fe877000640 1 Processor -- start 2024-03-27T06:15:35.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.445+0000 7fe877000640 1 -- start start 2024-03-27T06:15:35.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe877000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700c7ab0 0x7fe870072280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe877000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe8700727c0 0x7fe87015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe877000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87006caf0 con 0x7fe870154990 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe877000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fe8700c8c70 con 0x7fe8700727c0 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe877000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700c8f50 con 0x7fe8700c7ab0 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe86f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700c7ab0 0x7fe870072280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe86fe00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe870154990 0x7fe87006b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:56980/0 (socket says 172.21.15.132:56980) 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe86fe00640 1 -- 172.21.15.132:0/2067620493 learned_addr learned my addr 172.21.15.132:0/2067620493 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:35.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.446+0000 7fe875000640 1 --2- 172.21.15.132:0/2067620493 >> v2:172.21.15.134:3300/0 conn(0x7fe8700727c0 0x7fe87015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe86ea00640 1 -- 172.21.15.132:0/2067620493 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84007544 0 0) 0x7fe87006caf0 con 0x7fe870154990 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe86ea00640 1 -- 172.21.15.132:0/2067620493 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe854003520 con 0x7fe870154990 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe875000640 1 -- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700c7ab0 msgr2=0x7fe870072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe875000640 1 --2- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700c7ab0 0x7fe870072280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe875000640 1 -- 172.21.15.132:0/2067620493 >> v1:172.21.15.132:6789/0 conn(0x7fe870154990 legacy=0x7fe87006b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.447+0000 7fe875000640 1 -- 172.21.15.132:0/2067620493 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8700c9230 con 0x7fe8700727c0 2024-03-27T06:15:35.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.448+0000 7fe875000640 1 --2- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700727c0 0x7fe87015abb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe868009d90 tx=0x7fe86802bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af2eee0c2763b45a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:35.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe86ea00640 1 -- 172.21.15.132:0/2067620493 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868039070 con 0x7fe8700727c0 2024-03-27T06:15:35.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe86ea00640 1 -- 172.21.15.132:0/2067620493 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86802ccf0 con 0x7fe8700727c0 2024-03-27T06:15:35.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe86ea00640 1 -- 172.21.15.132:0/2067620493 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868034660 con 0x7fe8700727c0 2024-03-27T06:15:35.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 -- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700727c0 msgr2=0x7fe87015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 --2- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700727c0 0x7fe87015abb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe868009d90 tx=0x7fe86802bf40 comp rx=0 tx=0).stop 2024-03-27T06:15:35.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 -- 172.21.15.132:0/2067620493 shutdown_connections 2024-03-27T06:15:35.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 --2- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700727c0 0x7fe87015abb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 --2- 172.21.15.132:0/2067620493 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700c7ab0 0x7fe870072280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.449+0000 7fe877000640 1 -- 172.21.15.132:0/2067620493 >> 172.21.15.132:0/2067620493 conn(0x7fe87006e190 msgr2=0x7fe87006e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:35.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.450+0000 7fe877000640 1 -- 172.21.15.132:0/2067620493 shutdown_connections 2024-03-27T06:15:35.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.450+0000 7fe877000640 1 -- 172.21.15.132:0/2067620493 wait complete. 2024-03-27T06:15:35.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.452+0000 7fe877000640 1 Processor -- start 2024-03-27T06:15:35.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.452+0000 7fe877000640 1 -- start start 2024-03-27T06:15:35.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.453+0000 7fe877000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 0x7fe8700cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.453+0000 7fe877000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 0x7fe8701485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe877000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 0x7fe870148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe877000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe870154d70 con 0x7fe870154990 2024-03-27T06:15:35.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe877000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe8700c97f0 con 0x7fe8700c7ab0 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe877000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700c7eb0 con 0x7fe8700727c0 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 0x7fe8700cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 0x7fe8701485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 0x7fe8700cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:38420/0 (socket says 172.21.15.132:38420) 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86fe00640 1 -- 172.21.15.132:0/2698713741 learned_addr learned my addr 172.21.15.132:0/2698713741 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:35.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe875000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 0x7fe870148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 -- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 msgr2=0x7fe8700cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 0x7fe8700cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 -- 172.21.15.132:0/2698713741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 msgr2=0x7fe870148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 0x7fe870148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe86f400640 1 -- 172.21.15.132:0/2698713741 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87014e540 con 0x7fe8700c7ab0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.454+0000 7fe875000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 0x7fe870148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe86f400640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 0x7fe8701485d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00e8b0 tx=0x7fe85c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c017070 con 0x7fe8700c7ab0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c00c940 con 0x7fe8700c7ab0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8700cc6d0 con 0x7fe8700c7ab0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00cd30 con 0x7fe8700c7ab0 2024-03-27T06:15:35.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.455+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8700ccc90 con 0x7fe8700c7ab0 2024-03-27T06:15:35.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.456+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87006cd30 con 0x7fe8700c7ab0 2024-03-27T06:15:35.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.457+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe85c01a070 con 0x7fe8700c7ab0 2024-03-27T06:15:35.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.458+0000 7fe867e00640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 0x7fe85006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.458+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe85c08b6b0 con 0x7fe8700c7ab0 2024-03-27T06:15:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.459+0000 7fe86fe00640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 0x7fe85006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.459+0000 7fe86fe00640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 0x7fe85006e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe860005880 tx=0x7fe860009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.462+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe87006cd30 con 0x7fe8700c7ab0 2024-03-27T06:15:35.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.837+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe87006cad0 con 0x7fe8700c7ab0 2024-03-27T06:15:35.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.837+0000 7fe867e00640 1 -- 172.21.15.132:0/2698713741 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6053 (secure 0 0 0) 0x7fe85c059590 con 0x7fe8700c7ab0 2024-03-27T06:15:35.828 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:35.828 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:34.860918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:15:35.828 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 16 2024-03-27T06:15:35.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.839+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 msgr2=0x7fe85006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.839+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 0x7fe85006e9f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe860005880 tx=0x7fe860009040 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.839+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 msgr2=0x7fe8701485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.839+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 0x7fe8701485d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00e8b0 tx=0x7fe85c00ed70 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 shutdown_connections 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe870154990 0x7fe870148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe85006c530 0x7fe85006e9f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8700c7ab0 0x7fe8701485d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 --2- 172.21.15.132:0/2698713741 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8700727c0 0x7fe8700cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 >> 172.21.15.132:0/2698713741 conn(0x7fe87006e190 msgr2=0x7fe87014ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 shutdown_connections 2024-03-27T06:15:35.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:35.840+0000 7fe877000640 1 -- 172.21.15.132:0/2698713741 wait complete. 2024-03-27T06:15:35.852 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:37.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:38.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.046+0000 7f2461400640 1 Processor -- start 2024-03-27T06:15:38.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.046+0000 7f2461400640 1 -- start start 2024-03-27T06:15:38.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.047+0000 7f2461400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:38.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.047+0000 7f2461400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f245c071180 con 0x7f245c06c7a0 2024-03-27T06:15:38.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.047+0000 7f2461400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f245c071420 con 0x7f245c072780 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.047+0000 7f2461400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f245c0c6110 con 0x7f245c0c5d30 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:38436/0 (socket says 172.21.15.132:38436) 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 -- 172.21.15.132:0/996438384 learned_addr learned my addr 172.21.15.132:0/996438384 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 -- 172.21.15.132:0/996438384 >> v1:172.21.15.134:6789/0 conn(0x7f245c072780 legacy=0x7f245c06ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245b400640 1 -- 172.21.15.132:0/996438384 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194552680 0 0) 0x7f245c071420 con 0x7f245c072780 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 -- 172.21.15.132:0/996438384 >> v1:172.21.15.132:6789/0 conn(0x7f245c06c7a0 legacy=0x7f245c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.048+0000 7f245be00640 1 -- 172.21.15.132:0/996438384 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c0716e0 con 0x7f245c0c5d30 2024-03-27T06:15:38.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.049+0000 7f245be00640 1 --2- 172.21.15.132:0/996438384 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f244c0098a0 tx=0x7f244c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=def63b34f37cb183 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:38.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f245b400640 1 -- 172.21.15.132:0/996438384 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0317d0 con 0x7f245c0c5d30 2024-03-27T06:15:38.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f245b400640 1 -- 172.21.15.132:0/996438384 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244c037070 con 0x7f245c0c5d30 2024-03-27T06:15:38.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f245b400640 1 -- 172.21.15.132:0/996438384 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c031b90 con 0x7f245c0c5d30 2024-03-27T06:15:38.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 -- 172.21.15.132:0/996438384 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 msgr2=0x7f245c15ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 --2- 172.21.15.132:0/996438384 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f244c0098a0 tx=0x7f244c02b8f0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 -- 172.21.15.132:0/996438384 shutdown_connections 2024-03-27T06:15:38.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 --2- 172.21.15.132:0/996438384 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c15ec70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 -- 172.21.15.132:0/996438384 >> 172.21.15.132:0/996438384 conn(0x7f245c0cc7a0 msgr2=0x7f245c0c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:38.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 -- 172.21.15.132:0/996438384 shutdown_connections 2024-03-27T06:15:38.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.050+0000 7f2461400640 1 -- 172.21.15.132:0/996438384 wait complete. 2024-03-27T06:15:38.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.052+0000 7f2461400640 1 Processor -- start 2024-03-27T06:15:38.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.052+0000 7f2461400640 1 -- start start 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 0x7f245c1f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f245c072780 0x7f245c0bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c162420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f245c071f80 con 0x7f245c072780 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f245c071ca0 con 0x7f245c06c7a0 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2461400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f245c072260 con 0x7f245c0c5d30 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f245be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 0x7f245c1f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f245be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 0x7f245c1f50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:49346/0 (socket says 172.21.15.132:49346) 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f245be00640 1 -- 172.21.15.132:0/1366626207 learned_addr learned my addr 172.21.15.132:0/1366626207 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2453e00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f245c072780 0x7f245c0bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:38.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.053+0000 7f2460a00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:38.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 -- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 msgr2=0x7f245c1f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 0x7f245c1f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 -- 172.21.15.132:0/1366626207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f245c072780 msgr2=0x7f245c0bfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f245c072780 0x7f245c0bfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 -- 172.21.15.132:0/1366626207 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c15ba30 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2460a00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c162420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f245400c750 tx=0x7f245400cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454017070 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2454003ba0 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454005070 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245c1f45d0 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.054+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f245c15b700 con 0x7f245c0c5d30 2024-03-27T06:15:38.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.056+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f245c0686c0 con 0x7f245c0c5d30 2024-03-27T06:15:38.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.057+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f245401a070 con 0x7f245c0c5d30 2024-03-27T06:15:38.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.058+0000 7f2458c00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 0x7f243806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:38.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.058+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f245408ac60 con 0x7f245c0c5d30 2024-03-27T06:15:38.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.058+0000 7f245be00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 0x7f243806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:38.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.059+0000 7f245be00640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 0x7f243806e9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f244c0098a0 tx=0x7f244c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.062+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2454058bc0 con 0x7f245c0c5d30 2024-03-27T06:15:38.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.522+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f245c0c6460 con 0x7f245c0c5d30 2024-03-27T06:15:38.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.522+0000 7f2458c00640 1 -- 172.21.15.132:0/1366626207 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6053 (secure 0 0 0) 0x7f245405cec0 con 0x7f245c0c5d30 2024-03-27T06:15:38.513 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:38.513 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:34.860918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:15:38.513 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 16 2024-03-27T06:15:38.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 msgr2=0x7f243806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 0x7f243806e9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f244c0098a0 tx=0x7f244c03d040 comp rx=0 tx=0).stop 2024-03-27T06:15:38.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 msgr2=0x7f245c162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c162420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f245400c750 tx=0x7f245400cc10 comp rx=0 tx=0).stop 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 shutdown_connections 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f245c0c5d30 0x7f245c162420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f245c072780 0x7f245c0bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f243806c530 0x7f243806e9f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 --2- 172.21.15.132:0/1366626207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f245c06c7a0 0x7f245c1f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 >> 172.21.15.132:0/1366626207 conn(0x7f245c0cc7a0 msgr2=0x7f245c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 shutdown_connections 2024-03-27T06:15:38.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:38.526+0000 7f2461400640 1 -- 172.21.15.132:0/1366626207 wait complete. 2024-03-27T06:15:38.541 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:15:40.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:40.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.730+0000 7fa4fb200640 1 Processor -- start 2024-03-27T06:15:40.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.730+0000 7fa4fb200640 1 -- start start 2024-03-27T06:15:40.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4fb200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa4fc1549a0 0x7fa4fc0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4fb200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4fb200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4fc15de10 con 0x7fa4fc06be10 2024-03-27T06:15:40.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4fb200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa4fc0cada0 con 0x7fa4fc1549a0 2024-03-27T06:15:40.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4fb200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa4fc15e0b0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.731+0000 7fa4f9400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:38454/0 (socket says 172.21.15.132:38454) 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 -- 172.21.15.132:0/1007123080 learned_addr learned my addr 172.21.15.132:0/1007123080 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9e00640 1 --2- 172.21.15.132:0/1007123080 >> v2:172.21.15.132:3300/0 conn(0x7fa4fc1549a0 0x7fa4fc0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 -- 172.21.15.132:0/1007123080 >> v2:172.21.15.132:3300/0 conn(0x7fa4fc1549a0 msgr2=0x7fa4fc0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 --2- 172.21.15.132:0/1007123080 >> v2:172.21.15.132:3300/0 conn(0x7fa4fc1549a0 0x7fa4fc0cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 -- 172.21.15.132:0/1007123080 >> v1:172.21.15.134:6789/0 conn(0x7fa4fc06be10 legacy=0x7fa4fc15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.732+0000 7fa4f9400640 1 -- 172.21.15.132:0/1007123080 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc15e390 con 0x7fa4fc0cb870 2024-03-27T06:15:40.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.733+0000 7fa4f9400640 1 --2- 172.21.15.132:0/1007123080 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009870 tx=0x7fa4e802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=566c79a8fd9e0519 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:40.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.733+0000 7fa4f8a00640 1 -- 172.21.15.132:0/1007123080 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e802f830 con 0x7fa4fc0cb870 2024-03-27T06:15:40.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4f8a00640 1 -- 172.21.15.132:0/1007123080 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e802f9d0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4f8a00640 1 -- 172.21.15.132:0/1007123080 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e802fda0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 -- 172.21.15.132:0/1007123080 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 msgr2=0x7fa4fc06b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1007123080 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009870 tx=0x7fa4e802ba10 comp rx=0 tx=0).stop 2024-03-27T06:15:40.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 -- 172.21.15.132:0/1007123080 shutdown_connections 2024-03-27T06:15:40.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1007123080 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc0cb870 0x7fa4fc06b8d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1007123080 >> v2:172.21.15.132:3300/0 conn(0x7fa4fc1549a0 0x7fa4fc0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.734+0000 7fa4fb200640 1 -- 172.21.15.132:0/1007123080 >> 172.21.15.132:0/1007123080 conn(0x7fa4fc06d8d0 msgr2=0x7fa4fc06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:40.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.735+0000 7fa4fb200640 1 -- 172.21.15.132:0/1007123080 shutdown_connections 2024-03-27T06:15:40.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.735+0000 7fa4fb200640 1 -- 172.21.15.132:0/1007123080 wait complete. 2024-03-27T06:15:40.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.736+0000 7fa4fb200640 1 Processor -- start 2024-03-27T06:15:40.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.736+0000 7fa4fb200640 1 -- start start 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4fc06be10 0x7fa4fc153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc1549a0 0x7fa4fc0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc15ec30 con 0x7fa4fc06be10 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc15e950 con 0x7fa4fc0cb870 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fb200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa4fc15a820 con 0x7fa4fc1549a0 2024-03-27T06:15:40.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4f9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4f9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:49348/0 (socket says 172.21.15.132:49348) 2024-03-27T06:15:40.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4f9400640 1 -- 172.21.15.132:0/1712405627 learned_addr learned my addr 172.21.15.132:0/1712405627 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:40.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4fa800640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc1549a0 0x7fa4fc0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.737+0000 7fa4f9e00640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4fc06be10 0x7fa4fc153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 -- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc1549a0 msgr2=0x7fa4fc0cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc1549a0 0x7fa4fc0cf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 -- 172.21.15.132:0/1712405627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4fc06be10 msgr2=0x7fa4fc153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4fc06be10 0x7fa4fc153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 -- 172.21.15.132:0/1712405627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc153ee0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.738+0000 7fa4f9400640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8000c00 tx=0x7fa4e8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.739+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8037d10 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.739+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e8046070 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.739+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8041640 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.739+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4fc158620 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.739+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4fc1541a0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.740+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4fc0686c0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.742+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa4e8040070 con 0x7fa4fc0cb870 2024-03-27T06:15:40.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.743+0000 7fa4f2000640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 0x7fa4dc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:40.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.743+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa4e80adcf0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.743+0000 7fa4f9e00640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 0x7fa4dc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:40.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.744+0000 7fa4f9e00640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 0x7fa4dc06e9f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec006fd0 tx=0x7fa4ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:40.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.747+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa4e8033030 con 0x7fa4fc0cb870 2024-03-27T06:15:40.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.981+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4fc0b5550 con 0x7fa4fc0cb870 2024-03-27T06:15:40.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.982+0000 7fa4f2000640 1 -- 172.21.15.132:0/1712405627 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6058 (secure 0 0 0) 0x7fa4e807bbd0 con 0x7fa4fc0cb870 2024-03-27T06:15:40.972 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:40.972 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4241,4270,4273]},"id":1}]} 2024-03-27T06:15:40.973 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 17 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 msgr2=0x7fa4dc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 0x7fa4dc06e9f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec006fd0 tx=0x7fa4ec008040 comp rx=0 tx=0).stop 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 msgr2=0x7fa4fc0ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8000c00 tx=0x7fa4e8004730 comp rx=0 tx=0).stop 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 shutdown_connections 2024-03-27T06:15:40.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4fc1549a0 0x7fa4fc0cf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4dc06c530 0x7fa4dc06e9f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4fc0cb870 0x7fa4fc0ce840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 --2- 172.21.15.132:0/1712405627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4fc06be10 0x7fa4fc153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.984+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 >> 172.21.15.132:0/1712405627 conn(0x7fa4fc06d8d0 msgr2=0x7fa4fc159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.985+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 shutdown_connections 2024-03-27T06:15:40.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:40.985+0000 7fa4fb200640 1 -- 172.21.15.132:0/1712405627 wait complete. 2024-03-27T06:15:40.992 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=4244 2024-03-27T06:15:40.992 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2024-03-27T06:15:41.385 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:15:41.394+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-27T06:15:49.993 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-03-27T06:15:49.994 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-27T06:15:49.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-27T06:15:50.000 INFO:tasks.ceph.mds.c:Started 2024-03-27T06:15:50.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:50.216 INFO:tasks.ceph.mds.c.smithi132.stdout:starting mds.c at 2024-03-27T06:15:50.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.258+0000 7ff67dc00640 1 Processor -- start 2024-03-27T06:15:50.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.259+0000 7ff67dc00640 1 -- start start 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff67dc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 0x7ff67815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff67dc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff67806b520 0x7ff67806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff67dc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6780c1120 con 0x7ff678158f10 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff67dc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff67815fc20 con 0x7ff67806b520 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff67dc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff6780c13e0 con 0x7ff67806daf0 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff676a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff678158f10 0x7ff67806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:50302/0 (socket says 172.21.15.132:50302) 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff676a00640 1 -- 172.21.15.132:0/1932293156 learned_addr learned my addr 172.21.15.132:0/1932293156 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:15:50.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.260+0000 7ff66fe00640 1 --2- 172.21.15.132:0/1932293156 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 0x7ff67815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.261+0000 7ff676000640 1 -- 172.21.15.132:0/1932293156 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3849562777 0 0) 0x7ff6780c1120 con 0x7ff678158f10 2024-03-27T06:15:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.261+0000 7ff66fe00640 1 -- 172.21.15.132:0/1932293156 >> v2:172.21.15.132:3300/0 conn(0x7ff67806b520 msgr2=0x7ff67806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.261+0000 7ff66fe00640 1 --2- 172.21.15.132:0/1932293156 >> v2:172.21.15.132:3300/0 conn(0x7ff67806b520 0x7ff67806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.261+0000 7ff66fe00640 1 -- 172.21.15.132:0/1932293156 >> v1:172.21.15.134:6789/0 conn(0x7ff678158f10 legacy=0x7ff67806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.261+0000 7ff66fe00640 1 -- 172.21.15.132:0/1932293156 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6780c16c0 con 0x7ff67806daf0 2024-03-27T06:15:50.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.262+0000 7ff66fe00640 1 --2- 172.21.15.132:0/1932293156 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 0x7ff67815f6e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff6600097a0 tx=0x7ff66002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e313c9a6c3ffb60 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:50.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.262+0000 7ff676000640 1 -- 172.21.15.132:0/1932293156 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660035020 con 0x7ff67806daf0 2024-03-27T06:15:50.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.262+0000 7ff676000640 1 -- 172.21.15.132:0/1932293156 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff660031810 con 0x7ff67806daf0 2024-03-27T06:15:50.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.262+0000 7ff676000640 1 -- 172.21.15.132:0/1932293156 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660031be0 con 0x7ff67806daf0 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 -- 172.21.15.132:0/1932293156 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 msgr2=0x7ff67815f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 --2- 172.21.15.132:0/1932293156 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 0x7ff67815f6e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff6600097a0 tx=0x7ff66002b8b0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 -- 172.21.15.132:0/1932293156 shutdown_connections 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 --2- 172.21.15.132:0/1932293156 >> v2:172.21.15.132:3300/0 conn(0x7ff67806b520 0x7ff67806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 --2- 172.21.15.132:0/1932293156 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff67806daf0 0x7ff67815f6e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 -- 172.21.15.132:0/1932293156 >> 172.21.15.132:0/1932293156 conn(0x7ff6780cc060 msgr2=0x7ff6780cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:50.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.263+0000 7ff67dc00640 1 -- 172.21.15.132:0/1932293156 shutdown_connections 2024-03-27T06:15:50.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.264+0000 7ff67dc00640 1 -- 172.21.15.132:0/1932293156 wait complete. 2024-03-27T06:15:50.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.266+0000 7ff67dc00640 1 Processor -- start 2024-03-27T06:15:50.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.266+0000 7ff67dc00640 1 -- start start 2024-03-27T06:15:50.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 0x7ff678071470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 0x7ff6781f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 0x7ff6781f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780c1f60 con 0x7ff67806daf0 2024-03-27T06:15:50.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780c1c80 con 0x7ff67806b520 2024-03-27T06:15:50.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff67dc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff67806cec0 con 0x7ff678158f10 2024-03-27T06:15:50.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff676a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 0x7ff678071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:50.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff676a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 0x7ff678071470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:60556/0 (socket says 172.21.15.132:60556) 2024-03-27T06:15:50.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff676a00640 1 -- 172.21.15.132:0/3109173297 learned_addr learned my addr 172.21.15.132:0/3109173297 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:50.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff677400640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 0x7ff6781f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:50.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.268+0000 7ff66fe00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 0x7ff6781f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 -- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 msgr2=0x7ff678071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 0x7ff678071470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 -- 172.21.15.132:0/3109173297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 msgr2=0x7ff6781f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 0x7ff6781f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 -- 172.21.15.132:0/3109173297 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67814f4a0 con 0x7ff678158f10 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff66fe00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 0x7ff6781f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff677400640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 0x7ff6781f6130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff67000be20 tx=0x7ff67000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670019070 con 0x7ff678158f10 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff670014440 con 0x7ff678158f10 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.269+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670014830 con 0x7ff678158f10 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.270+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67815e6a0 con 0x7ff678158f10 2024-03-27T06:15:50.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.270+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67815dd70 con 0x7ff678158f10 2024-03-27T06:15:50.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.271+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff67001c070 con 0x7ff678158f10 2024-03-27T06:15:50.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.272+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff678068840 con 0x7ff678158f10 2024-03-27T06:15:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.272+0000 7ff66ea00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 0x7ff65406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.272+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff67008b630 con 0x7ff678158f10 2024-03-27T06:15:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.273+0000 7ff676a00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 0x7ff65406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:50.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.273+0000 7ff676a00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 0x7ff65406ea20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff668000f60 tx=0x7ff668007e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:50.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.278+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff67814f4a0 con 0x7ff678158f10 2024-03-27T06:15:50.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.716+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff67806c570 con 0x7ff678158f10 2024-03-27T06:15:50.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.717+0000 7ff66ea00640 1 -- 172.21.15.132:0/3109173297 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6058 (secure 0 0 0) 0x7ff670059510 con 0x7ff678158f10 2024-03-27T06:15:50.709 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:50.709 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4241,4270,4273]},"id":1}]} 2024-03-27T06:15:50.709 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 17 2024-03-27T06:15:50.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 msgr2=0x7ff65406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 0x7ff65406ea20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff668000f60 tx=0x7ff668007e10 comp rx=0 tx=0).stop 2024-03-27T06:15:50.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 msgr2=0x7ff6781f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:50.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 0x7ff6781f6130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff67000be20 tx=0x7ff67000bf10 comp rx=0 tx=0).stop 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 shutdown_connections 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff678158f10 0x7ff6781f6130 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67806daf0 0x7ff6781f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff65406c560 0x7ff65406ea20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 --2- 172.21.15.132:0/3109173297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff67806b520 0x7ff678071470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.720+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 >> 172.21.15.132:0/3109173297 conn(0x7ff6780cc060 msgr2=0x7ff6780cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.721+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 shutdown_connections 2024-03-27T06:15:50.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:50.721+0000 7ff67dc00640 1 -- 172.21.15.132:0/3109173297 wait complete. 2024-03-27T06:15:50.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-27T06:15:52.731 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:15:52.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.962+0000 7f08ebe00640 1 Processor -- start 2024-03-27T06:15:52.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.962+0000 7f08ebe00640 1 -- start start 2024-03-27T06:15:52.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08ebe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f08ec1549a0 0x7f08ec0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08ebe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec15adb0 0x7f08ec15b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08ebe00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec15de10 con 0x7f08ec0cb870 2024-03-27T06:15:52.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08ebe00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f08ec0cada0 con 0x7f08ec1549a0 2024-03-27T06:15:52.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08ebe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec15e0b0 con 0x7f08ec15adb0 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08eaa00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f08ec1549a0 0x7f08ec0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.964+0000 7f08eaa00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f08ec1549a0 0x7f08ec0cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:32876/0 (socket says 172.21.15.132:32876) 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eaa00640 1 -- 172.21.15.132:0/1834193035 learned_addr learned my addr 172.21.15.132:0/1834193035 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eb400640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec15adb0 0x7f08ec15b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eaa00640 1 -- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec15adb0 msgr2=0x7f08ec15b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eaa00640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec15adb0 0x7f08ec15b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eaa00640 1 -- 172.21.15.132:0/1834193035 >> v1:172.21.15.134:6789/0 conn(0x7f08ec0cb870 legacy=0x7f08ec06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08eaa00640 1 -- 172.21.15.132:0/1834193035 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec15e390 con 0x7f08ec1549a0 2024-03-27T06:15:52.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.965+0000 7f08e9600640 1 -- 172.21.15.132:0/1834193035 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979446586 0 0) 0x7f08ec15de10 con 0x7f08ec0cb870 2024-03-27T06:15:52.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.967+0000 7f08eaa00640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec0cd6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009870 tx=0x7f08dc02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966f12fa52d411c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:52.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.967+0000 7f08e9600640 1 -- 172.21.15.132:0/1834193035 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc02f830 con 0x7f08ec1549a0 2024-03-27T06:15:52.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.967+0000 7f08e9600640 1 -- 172.21.15.132:0/1834193035 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc02f9d0 con 0x7f08ec1549a0 2024-03-27T06:15:52.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.967+0000 7f08e9600640 1 -- 172.21.15.132:0/1834193035 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc02fda0 con 0x7f08ec1549a0 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 -- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 msgr2=0x7f08ec0cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec0cd6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009870 tx=0x7f08dc02ba10 comp rx=0 tx=0).stop 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 -- 172.21.15.132:0/1834193035 shutdown_connections 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec15adb0 0x7f08ec15b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 --2- 172.21.15.132:0/1834193035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec0cd6a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 -- 172.21.15.132:0/1834193035 >> 172.21.15.132:0/1834193035 conn(0x7f08ec06d8d0 msgr2=0x7f08ec06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 -- 172.21.15.132:0/1834193035 shutdown_connections 2024-03-27T06:15:52.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.968+0000 7f08ebe00640 1 -- 172.21.15.132:0/1834193035 wait complete. 2024-03-27T06:15:52.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.971+0000 7f08ebe00640 1 Processor -- start 2024-03-27T06:15:52.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.971+0000 7f08ebe00640 1 -- start start 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08ec15adb0 0x7f08ec0d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec15ec30 con 0x7f08ec15adb0 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec15e950 con 0x7f08ec1549a0 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.972+0000 7f08ebe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec15b700 con 0x7f08ec0cb870 2024-03-27T06:15:52.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:52.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:47978/0 (socket says 172.21.15.132:47978) 2024-03-27T06:15:52.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 -- 172.21.15.132:0/4106514635 learned_addr learned my addr 172.21.15.132:0/4106514635 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:15:52.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08ea000640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:52.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 -- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 msgr2=0x7f08ec1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:52.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec1f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 -- 172.21.15.132:0/4106514635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08ec15adb0 msgr2=0x7f08ec0d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08ec15adb0 0x7f08ec0d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.973+0000 7f08eaa00640 1 -- 172.21.15.132:0/4106514635 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec0c73c0 con 0x7f08ec0cb870 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.974+0000 7f08eaa00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f08dc000c00 tx=0x7f08dc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.974+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc037d10 con 0x7f08ec0cb870 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.974+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc046070 con 0x7f08ec0cb870 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.974+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc041640 con 0x7f08ec0cb870 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.974+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec1f06d0 con 0x7f08ec0cb870 2024-03-27T06:15:52.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.975+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec1f43d0 con 0x7f08ec0cb870 2024-03-27T06:15:52.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.976+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec0686c0 con 0x7f08ec0cb870 2024-03-27T06:15:52.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.977+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f08dc040070 con 0x7f08ec0cb870 2024-03-27T06:15:52.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.979+0000 7f08e2a00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 0x7f08cc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:15:52.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.979+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f08dc0add50 con 0x7f08ec0cb870 2024-03-27T06:15:52.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.979+0000 7f08ea000640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 0x7f08cc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:15:52.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.980+0000 7f08ea000640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 0x7f08cc06e9f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0d1020 tx=0x7f08d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:15:52.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:52.987+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f08dc033030 con 0x7f08ec0cb870 2024-03-27T06:15:53.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.429+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08ec158a30 con 0x7f08ec0cb870 2024-03-27T06:15:53.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.430+0000 7f08e2a00640 1 -- 172.21.15.132:0/4106514635 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f08dc07c300 con 0x7f08ec0cb870 2024-03-27T06:15:53.422 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:15:53.422 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:15:53.422 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:15:53.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.432+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 msgr2=0x7f08cc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.432+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 0x7f08cc06e9f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0d1020 tx=0x7f08d8009040 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.432+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 msgr2=0x7f08ec144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.432+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f08dc000c00 tx=0x7f08dc004730 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 shutdown_connections 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08ec15adb0 0x7f08ec0d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f08cc06c530 0x7f08cc06e9f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08ec1549a0 0x7f08ec1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 --2- 172.21.15.132:0/4106514635 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08ec0cb870 0x7f08ec144ec0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 >> 172.21.15.132:0/4106514635 conn(0x7f08ec06d8d0 msgr2=0x7f08ec158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 shutdown_connections 2024-03-27T06:15:53.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:15:53.433+0000 7f08ebe00640 1 -- 172.21.15.132:0/4106514635 wait complete. 2024-03-27T06:15:53.450 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-03-27T06:15:53.450 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2024-03-27T06:16:03.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:16:03.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:16:05.387 INFO:tasks.workunit.client.0.smithi132.stderr:Updating files: 62% (7647/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-27T06:16:06.092 INFO:tasks.workunit.client.0.smithi132.stderr:Note: switching to '155268c4e432a12433aa833f174f9fe3b1016ae0'. 2024-03-27T06:16:06.092 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.092 INFO:tasks.workunit.client.0.smithi132.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-27T06:16:06.092 INFO:tasks.workunit.client.0.smithi132.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-27T06:16:06.092 INFO:tasks.workunit.client.0.smithi132.stderr:state without impacting any branches by switching back to a branch. 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: git switch -c 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr:Or undo this operation with: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: git switch - 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr: 2024-03-27T06:16:06.093 INFO:tasks.workunit.client.0.smithi132.stderr:HEAD is now at 155268c4e43 Merge PR #51993 into wip-rishabh-testing-20240326.131558 2024-03-27T06:16:06.103 DEBUG:teuthology.orchestra.run.smithi132:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-27T06:16:06.164 INFO:tasks.workunit.client.0.smithi132.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-27T06:16:06.171 INFO:tasks.workunit.client.0.smithi132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-27T06:16:06.171 INFO:tasks.workunit.client.0.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-27T06:16:07.006 INFO:tasks.workunit.client.0.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-27T06:16:07.121 INFO:tasks.workunit.client.0.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-27T06:16:07.207 INFO:tasks.workunit.client.0.smithi132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-27T06:16:07.212 INFO:tasks.workunit.client.0.smithi132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-27T06:16:07.212 INFO:tasks.workunit.client.0.smithi132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-27T06:16:07.293 INFO:tasks.workunit.client.0.smithi132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-27T06:16:07.303 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:16:07.303 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-27T06:16:07.352 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-03-27T06:16:07.353 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-03-27T06:16:07.353 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-27T06:16:07.462 INFO:tasks.workunit.client.0.smithi132.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-03-27T06:16:12.994 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir foo 2024-03-27T06:16:13.018 INFO:tasks.workunit.client.0.smithi132.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-03-27T06:16:13.163 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir foo/.snap/barsnap 2024-03-27T06:16:13.196 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-03-27T06:16:13.212 INFO:tasks.workunit.client.0.smithi132.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-03-27T06:16:13.652 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'passed: files are identical' 2024-03-27T06:16:13.653 INFO:tasks.workunit.client.0.smithi132.stdout:passed: files are identical 2024-03-27T06:16:13.653 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir foo/.snap/barsnap 2024-03-27T06:16:13.726 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:16:13.727 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:16:13.727 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:16:13.727 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:16:13.821 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-03-27T06:16:13.821 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-27T06:16:18.598 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 1 2024-03-27T06:16:18.600 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 2 2024-03-27T06:16:18.601 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 3 2024-03-27T06:16:18.604 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' -e file4 ']' 2024-03-27T06:16:18.604 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir .snap/snap1 2024-03-27T06:16:18.662 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 4 2024-03-27T06:16:18.665 INFO:tasks.workunit.client.0.smithi132.stderr:++ ls 2024-03-27T06:16:18.670 INFO:tasks.workunit.client.0.smithi132.stderr:+ now='file1 2024-03-27T06:16:18.670 INFO:tasks.workunit.client.0.smithi132.stderr:file2 2024-03-27T06:16:18.670 INFO:tasks.workunit.client.0.smithi132.stderr:file3 2024-03-27T06:16:18.670 INFO:tasks.workunit.client.0.smithi132.stderr:file4' 2024-03-27T06:16:18.671 INFO:tasks.workunit.client.0.smithi132.stderr:++ ls .snap/snap1 2024-03-27T06:16:18.681 INFO:tasks.workunit.client.0.smithi132.stderr:+ then='file1 2024-03-27T06:16:18.681 INFO:tasks.workunit.client.0.smithi132.stderr:file2 2024-03-27T06:16:18.681 INFO:tasks.workunit.client.0.smithi132.stderr:file3' 2024-03-27T06:16:18.681 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir .snap/snap1 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' 'file1 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:file2 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:file3 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:file4' = 'file1 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:file2 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:file3' ']' 2024-03-27T06:16:18.688 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 1 2024-03-27T06:16:18.698 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 2 2024-03-27T06:16:18.705 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 3 2024-03-27T06:16:18.716 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir .snap/snap1 2024-03-27T06:16:18.745 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 4 2024-03-27T06:16:18.751 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir .snap/snap1 2024-03-27T06:16:18.762 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm file1 file2 file3 file4 2024-03-27T06:16:18.772 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:16:18.773 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:16:18.774 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:16:18.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:16:18.865 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-03-27T06:16:18.865 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-27T06:16:23.581 INFO:tasks.workunit.client.0.smithi132.stdout:Create dir 100 to 199 ... 2024-03-27T06:16:23.600 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 100 2024-03-27T06:16:23.623 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 101 2024-03-27T06:16:23.649 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 102 2024-03-27T06:16:23.677 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 103 2024-03-27T06:16:23.705 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 104 2024-03-27T06:16:23.731 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 105 2024-03-27T06:16:23.757 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 106 2024-03-27T06:16:23.780 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 107 2024-03-27T06:16:23.801 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 108 2024-03-27T06:16:23.827 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 109 2024-03-27T06:16:23.856 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 110 2024-03-27T06:16:23.878 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 111 2024-03-27T06:16:23.900 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 112 2024-03-27T06:16:23.924 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 113 2024-03-27T06:16:23.950 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 114 2024-03-27T06:16:23.978 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 115 2024-03-27T06:16:24.002 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 116 2024-03-27T06:16:24.023 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 117 2024-03-27T06:16:24.046 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 118 2024-03-27T06:16:24.068 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 119 2024-03-27T06:16:24.090 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 120 2024-03-27T06:16:24.110 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 121 2024-03-27T06:16:24.136 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 122 2024-03-27T06:16:24.161 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 123 2024-03-27T06:16:24.182 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 124 2024-03-27T06:16:24.201 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 125 2024-03-27T06:16:24.220 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 126 2024-03-27T06:16:24.241 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 127 2024-03-27T06:16:24.265 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 128 2024-03-27T06:16:24.287 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 129 2024-03-27T06:16:24.311 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 130 2024-03-27T06:16:24.333 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 131 2024-03-27T06:16:24.357 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 132 2024-03-27T06:16:24.380 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 133 2024-03-27T06:16:24.408 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 134 2024-03-27T06:16:24.438 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 135 2024-03-27T06:16:24.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:24.469 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 136 2024-03-27T06:16:24.492 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 137 2024-03-27T06:16:24.515 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 138 2024-03-27T06:16:24.535 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 139 2024-03-27T06:16:24.560 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 140 2024-03-27T06:16:24.577 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 141 2024-03-27T06:16:24.594 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 142 2024-03-27T06:16:24.610 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 143 2024-03-27T06:16:24.626 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 144 2024-03-27T06:16:24.644 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 145 2024-03-27T06:16:24.662 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 146 2024-03-27T06:16:24.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.689+0000 7f5bfae00640 1 Processor -- start 2024-03-27T06:16:24.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.689+0000 7f5bfae00640 1 -- start start 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.691+0000 7f5bfae00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 0x7f5bf415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.691+0000 7f5bfae00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 0x7f5bf40ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.691+0000 7f5bfae00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bf4158670 con 0x7f5bf40c9c80 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.691+0000 7f5bfae00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf406ed40 con 0x7f5bf4158290 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.691+0000 7f5bfae00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf406e2f0 con 0x7f5bf415b120 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf3e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 0x7f5bf415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf8e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5bf40c9c80 0x7f5bf415e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:36766/0 (socket says 172.21.15.132:36766) 2024-03-27T06:16:24.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5beb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 0x7f5bf40ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf8e00640 1 -- 172.21.15.132:0/2024194280 learned_addr learned my addr 172.21.15.132:0/2024194280 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:24.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf3400640 1 -- 172.21.15.132:0/2024194280 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667597119 0 0) 0x7f5bf4158670 con 0x7f5bf40c9c80 2024-03-27T06:16:24.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf3400640 1 -- 172.21.15.132:0/2024194280 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bd8003520 con 0x7f5bf40c9c80 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5beb400640 1 -- 172.21.15.132:0/2024194280 >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 msgr2=0x7f5bf415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5beb400640 1 --2- 172.21.15.132:0/2024194280 >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 0x7f5bf415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5beb400640 1 -- 172.21.15.132:0/2024194280 >> v1:172.21.15.134:6789/0 conn(0x7f5bf40c9c80 legacy=0x7f5bf415e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5beb400640 1 -- 172.21.15.132:0/2024194280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf406c6d0 con 0x7f5bf415b120 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.692+0000 7f5bf3e00640 1 --2- 172.21.15.132:0/2024194280 >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 0x7f5bf415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:24.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.693+0000 7f5beb400640 1 --2- 172.21.15.132:0/2024194280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 0x7f5bf40ce290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0097a0 tx=0x7f5bdc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b4acaaa2972f84 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bf3400640 1 -- 172.21.15.132:0/2024194280 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc035020 con 0x7f5bf415b120 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bf3400640 1 -- 172.21.15.132:0/2024194280 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bdc031810 con 0x7f5bf415b120 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bf3400640 1 -- 172.21.15.132:0/2024194280 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc031be0 con 0x7f5bf415b120 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 -- 172.21.15.132:0/2024194280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 msgr2=0x7f5bf40ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 --2- 172.21.15.132:0/2024194280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 0x7f5bf40ce290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0097a0 tx=0x7f5bdc02b8b0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 -- 172.21.15.132:0/2024194280 shutdown_connections 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 --2- 172.21.15.132:0/2024194280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf415b120 0x7f5bf40ce290 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 --2- 172.21.15.132:0/2024194280 >> v2:172.21.15.134:3300/0 conn(0x7f5bf4158290 0x7f5bf415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 -- 172.21.15.132:0/2024194280 >> 172.21.15.132:0/2024194280 conn(0x7f5bf4067280 msgr2=0x7f5bf4067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:24.686 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 147 2024-03-27T06:16:24.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 -- 172.21.15.132:0/2024194280 shutdown_connections 2024-03-27T06:16:24.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.694+0000 7f5bfae00640 1 -- 172.21.15.132:0/2024194280 wait complete. 2024-03-27T06:16:24.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.696+0000 7f5bfae00640 1 Processor -- start 2024-03-27T06:16:24.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.696+0000 7f5bfae00640 1 -- start start 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bf4158290 0x7f5bf41548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 0x7f5bf40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf406e5d0 con 0x7f5bf4158290 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf406f690 con 0x7f5bf40c9c80 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.697+0000 7f5bfae00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf406f1d0 con 0x7f5bf41544c0 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf8e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 0x7f5bf40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:52294/0 (socket says 172.21.15.132:52294) 2024-03-27T06:16:24.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 -- 172.21.15.132:0/3416885653 learned_addr learned my addr 172.21.15.132:0/3416885653 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:24.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5beb400640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bf4158290 0x7f5bf41548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 -- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 msgr2=0x7f5bf40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 0x7f5bf40cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 -- 172.21.15.132:0/3416885653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bf4158290 msgr2=0x7f5bf41548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bf4158290 0x7f5bf41548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 -- 172.21.15.132:0/3416885653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf41f4510 con 0x7f5bf40c9c80 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf8e00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 0x7f5bf40cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.698+0000 7f5bf3e00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5be400efc0 tx=0x7f5be400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.699+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4018020 con 0x7f5bf40c9c80 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.699+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be4014680 con 0x7f5bf40c9c80 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.699+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4014a50 con 0x7f5bf40c9c80 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.699+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf40cc280 con 0x7f5bf40c9c80 2024-03-27T06:16:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.699+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf41f47b0 con 0x7f5bf40c9c80 2024-03-27T06:16:24.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.700+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf4068840 con 0x7f5bf40c9c80 2024-03-27T06:16:24.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.702+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5be4005d10 con 0x7f5bf40c9c80 2024-03-27T06:16:24.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.703+0000 7f5bf0c00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 0x7f5bd006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:24.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.703+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7f5be408c390 con 0x7f5bf40c9c80 2024-03-27T06:16:24.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.703+0000 7f5beb400640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 0x7f5bd006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:24.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.704+0000 7f5beb400640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 0x7f5bd006ea40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc036000 tx=0x7f5bdc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:24.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.710+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5be4022030 con 0x7f5bf40c9c80 2024-03-27T06:16:24.705 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 148 2024-03-27T06:16:24.720 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 149 2024-03-27T06:16:24.734 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 150 2024-03-27T06:16:24.748 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 151 2024-03-27T06:16:24.762 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 152 2024-03-27T06:16:24.777 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 153 2024-03-27T06:16:24.793 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 154 2024-03-27T06:16:24.812 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 155 2024-03-27T06:16:24.838 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 156 2024-03-27T06:16:24.860 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 157 2024-03-27T06:16:24.882 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 158 2024-03-27T06:16:24.905 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 159 2024-03-27T06:16:24.923 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 160 2024-03-27T06:16:24.943 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 161 2024-03-27T06:16:24.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.978+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bf4155d80 con 0x7f5bf40c9c80 2024-03-27T06:16:24.969 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 162 2024-03-27T06:16:24.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.979+0000 7f5bf0c00640 1 -- 172.21.15.132:0/3416885653 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f5be4010040 con 0x7f5bf40c9c80 2024-03-27T06:16:24.970 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:24.970 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:24.970 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:24.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 msgr2=0x7f5bd006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 0x7f5bd006ea40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc036000 tx=0x7f5bdc03c040 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 msgr2=0x7f5bf40b2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5be400efc0 tx=0x7f5be400c490 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 shutdown_connections 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bf41544c0 0x7f5bf40cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bf4158290 0x7f5bf41548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5bd006c580 0x7f5bd006ea40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 --2- 172.21.15.132:0/3416885653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bf40c9c80 0x7f5bf40b2540 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 >> 172.21.15.132:0/3416885653 conn(0x7f5bf4067280 msgr2=0x7f5bf40c4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 shutdown_connections 2024-03-27T06:16:24.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:24.982+0000 7f5bfae00640 1 -- 172.21.15.132:0/3416885653 wait complete. 2024-03-27T06:16:24.991 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 163 2024-03-27T06:16:24.993 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2024-03-27T06:16:24.993 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-03-27T06:16:24.993 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:16:25.014 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 164 2024-03-27T06:16:25.038 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 165 2024-03-27T06:16:25.040 INFO:tasks.ceph.mds.e:Stopped 2024-03-27T06:16:25.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:25.058 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 166 2024-03-27T06:16:25.076 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 167 2024-03-27T06:16:25.093 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 168 2024-03-27T06:16:25.109 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 169 2024-03-27T06:16:25.127 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 170 2024-03-27T06:16:25.144 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 171 2024-03-27T06:16:25.163 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 172 2024-03-27T06:16:25.183 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 173 2024-03-27T06:16:25.202 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 174 2024-03-27T06:16:25.220 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 175 2024-03-27T06:16:25.238 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 176 2024-03-27T06:16:25.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.266+0000 7fd398c00640 1 Processor -- start 2024-03-27T06:16:25.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.266+0000 7fd398c00640 1 -- start start 2024-03-27T06:16:25.256 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 177 2024-03-27T06:16:25.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd398c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd3941590a0 0x7fd394071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:25.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd398c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39406b630 con 0x7fd3940716b0 2024-03-27T06:16:25.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd398c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39406b8d0 con 0x7fd394152950 2024-03-27T06:16:25.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd398c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fd394152d30 con 0x7fd3941590a0 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd392800640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd3940716b0 0x7fd39406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54420/0 (socket says 172.21.15.132:54420) 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd392800640 1 -- 172.21.15.132:0/107080971 learned_addr learned my addr 172.21.15.132:0/107080971 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.268+0000 7fd391400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd3941590a0 0x7fd394071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.269+0000 7fd390a00640 1 -- 172.21.15.132:0/107080971 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101707057 0 0) 0x7fd39406b630 con 0x7fd3940716b0 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.269+0000 7fd391400640 1 -- 172.21.15.132:0/107080971 >> v1:172.21.15.134:6789/0 conn(0x7fd394152950 legacy=0x7fd394073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.269+0000 7fd391400640 1 -- 172.21.15.132:0/107080971 >> v1:172.21.15.132:6789/0 conn(0x7fd3940716b0 legacy=0x7fd39406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.269+0000 7fd391400640 1 -- 172.21.15.132:0/107080971 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39406bbb0 con 0x7fd3941590a0 2024-03-27T06:16:25.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.270+0000 7fd391400640 1 --2- 172.21.15.132:0/107080971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd394071170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd380009720 tx=0x7fd38002b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c7b21c13dde254d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:25.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.270+0000 7fd390a00640 1 -- 172.21.15.132:0/107080971 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380037070 con 0x7fd3941590a0 2024-03-27T06:16:25.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.270+0000 7fd390a00640 1 -- 172.21.15.132:0/107080971 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd380030710 con 0x7fd3941590a0 2024-03-27T06:16:25.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.270+0000 7fd390a00640 1 -- 172.21.15.132:0/107080971 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380030ae0 con 0x7fd3941590a0 2024-03-27T06:16:25.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 -- 172.21.15.132:0/107080971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 msgr2=0x7fd394071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 --2- 172.21.15.132:0/107080971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd394071170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd380009720 tx=0x7fd38002b730 comp rx=0 tx=0).stop 2024-03-27T06:16:25.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 -- 172.21.15.132:0/107080971 shutdown_connections 2024-03-27T06:16:25.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 --2- 172.21.15.132:0/107080971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd394071170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 -- 172.21.15.132:0/107080971 >> 172.21.15.132:0/107080971 conn(0x7fd3940cc060 msgr2=0x7fd3940cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:25.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 -- 172.21.15.132:0/107080971 shutdown_connections 2024-03-27T06:16:25.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.271+0000 7fd398c00640 1 -- 172.21.15.132:0/107080971 wait complete. 2024-03-27T06:16:25.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.272+0000 7fd398c00640 1 Processor -- start 2024-03-27T06:16:25.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 -- start start 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 0x7fd3940c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394152950 0x7fd3941ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd39406c450 con 0x7fd394152950 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd39406c170 con 0x7fd3941590a0 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.273+0000 7fd398c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd3941594c0 con 0x7fd3940716b0 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:25.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd391e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 0x7fd3940c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:25.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:52304/0 (socket says 172.21.15.132:52304) 2024-03-27T06:16:25.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 -- 172.21.15.132:0/3946659822 learned_addr learned my addr 172.21.15.132:0/3946659822 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:25.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd391e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 0x7fd3940c5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59008/0 (socket says 172.21.15.132:59008) 2024-03-27T06:16:25.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd391400640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394152950 0x7fd3941ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 -- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 msgr2=0x7fd3940c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 0x7fd3940c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 -- 172.21.15.132:0/3946659822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394152950 msgr2=0x7fd3941ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394152950 0x7fd3941ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 -- 172.21.15.132:0/3946659822 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3941501b0 con 0x7fd3941590a0 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.274+0000 7fd392800640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00c7a0 tx=0x7fd38c00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.275+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c018070 con 0x7fd3941590a0 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.275+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd38c00a640 con 0x7fd3941590a0 2024-03-27T06:16:25.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.275+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c00aab0 con 0x7fd3941590a0 2024-03-27T06:16:25.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.275+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd394150490 con 0x7fd3941590a0 2024-03-27T06:16:25.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.275+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3940c4930 con 0x7fd3941590a0 2024-03-27T06:16:25.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.276+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd394068840 con 0x7fd3941590a0 2024-03-27T06:16:25.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.277+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd38c00ac50 con 0x7fd3941590a0 2024-03-27T06:16:25.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.278+0000 7fd38a000640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 0x7fd37406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:25.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.278+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7fd38c08adf0 con 0x7fd3941590a0 2024-03-27T06:16:25.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.278+0000 7fd391e00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 0x7fd37406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:25.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.278+0000 7fd391e00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 0x7fd37406e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd384000f60 tx=0x7fd384009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:25.275 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 178 2024-03-27T06:16:25.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.285+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd38c058c90 con 0x7fd3941590a0 2024-03-27T06:16:25.291 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 179 2024-03-27T06:16:25.306 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 180 2024-03-27T06:16:25.322 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 181 2024-03-27T06:16:25.337 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 182 2024-03-27T06:16:25.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.528+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3940cb320 con 0x7fd3941590a0 2024-03-27T06:16:25.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.529+0000 7fd38a000640 1 -- 172.21.15.132:0/3946659822 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7fd38c05cf90 con 0x7fd3941590a0 2024-03-27T06:16:25.520 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:25.520 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:25.520 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:25.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 msgr2=0x7fd37406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 0x7fd37406e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd384000f60 tx=0x7fd384009040 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 msgr2=0x7fd39414fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00c7a0 tx=0x7fd38c00cc60 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 shutdown_connections 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd37406c530 0x7fd37406e9f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3941590a0 0x7fd39414fa20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394152950 0x7fd3941ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 --2- 172.21.15.132:0/3946659822 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd3940716b0 0x7fd3940c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 >> 172.21.15.132:0/3946659822 conn(0x7fd3940cc060 msgr2=0x7fd3941509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 shutdown_connections 2024-03-27T06:16:25.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:25.532+0000 7fd398c00640 1 -- 172.21.15.132:0/3946659822 wait complete. 2024-03-27T06:16:25.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:25.738 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 183 2024-03-27T06:16:25.760 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 184 2024-03-27T06:16:25.785 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 185 2024-03-27T06:16:25.806 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 186 2024-03-27T06:16:25.829 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 187 2024-03-27T06:16:25.853 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 188 2024-03-27T06:16:25.877 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 189 2024-03-27T06:16:25.900 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 190 2024-03-27T06:16:25.929 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 191 2024-03-27T06:16:25.956 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 192 2024-03-27T06:16:25.986 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 193 2024-03-27T06:16:26.015 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 194 2024-03-27T06:16:26.039 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 195 2024-03-27T06:16:26.062 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 196 2024-03-27T06:16:26.086 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 197 2024-03-27T06:16:26.109 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 198 2024-03-27T06:16:26.132 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 199 2024-03-27T06:16:26.162 INFO:tasks.workunit.client.0.smithi132.stdout:Take first snapshot .snap/test1 2024-03-27T06:16:27.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:27.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.764+0000 7f2ef0a00640 1 Processor -- start 2024-03-27T06:16:27.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.764+0000 7f2ef0a00640 1 -- start start 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.765+0000 7f2ef0a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2eec151830 0x7f2eec1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.765+0000 7f2ef0a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec157ad0 con 0x7f2eec0c68f0 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.765+0000 7f2ef0a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec157db0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2ef0a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2eec071290 con 0x7f2eec151830 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2ee2a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2eec0c64d0 0x7f2eec0b1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:36790/0 (socket says 172.21.15.132:36790) 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2ee2a00640 1 -- 172.21.15.132:0/2526004728 learned_addr learned my addr 172.21.15.132:0/2526004728 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:27.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeb400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2eec151830 0x7f2eec1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeaa00640 1 -- 172.21.15.132:0/2526004728 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 59389559 0 0) 0x7f2eec157db0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeaa00640 1 -- 172.21.15.132:0/2526004728 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ed0003520 con 0x7f2eec0c64d0 2024-03-27T06:16:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeb400640 1 -- 172.21.15.132:0/2526004728 >> v1:172.21.15.134:6789/0 conn(0x7f2eec0c64d0 legacy=0x7f2eec0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeb400640 1 -- 172.21.15.132:0/2526004728 >> v1:172.21.15.132:6789/0 conn(0x7f2eec0c68f0 legacy=0x7f2eec0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:27.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.766+0000 7f2eeb400640 1 -- 172.21.15.132:0/2526004728 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec158090 con 0x7f2eec151830 2024-03-27T06:16:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.767+0000 7f2eeb400640 1 --2- 172.21.15.132:0/2526004728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec151830 0x7f2eec1574f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5868421f67fa9ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.767+0000 7f2eeaa00640 1 -- 172.21.15.132:0/2526004728 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc02f810 con 0x7f2eec151830 2024-03-27T06:16:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2eeaa00640 1 -- 172.21.15.132:0/2526004728 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc02f9b0 con 0x7f2eec151830 2024-03-27T06:16:27.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2eeaa00640 1 -- 172.21.15.132:0/2526004728 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc02fd80 con 0x7f2eec151830 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2526004728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec151830 msgr2=0x7f2eec1574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2526004728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec151830 0x7f2eec1574f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02b9f0 comp rx=0 tx=0).stop 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2526004728 shutdown_connections 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2526004728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec151830 0x7f2eec1574f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2526004728 >> 172.21.15.132:0/2526004728 conn(0x7f2eec0cc7a0 msgr2=0x7f2eec0c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2526004728 shutdown_connections 2024-03-27T06:16:27.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.768+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2526004728 wait complete. 2024-03-27T06:16:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.770+0000 7f2ef0a00640 1 Processor -- start 2024-03-27T06:16:27.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.770+0000 7f2ef0a00640 1 -- start start 2024-03-27T06:16:27.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2eec151830 0x7f2eec155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec072b90 con 0x7f2eec151830 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0728b0 con 0x7f2eec0c68f0 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ef0a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2eec151c10 con 0x7f2eec0c64d0 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ee2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2eebe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2eec151830 0x7f2eec155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2eeb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2eeb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59016/0 (socket says 172.21.15.132:59016) 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2eeb400640 1 -- 172.21.15.132:0/2886638903 learned_addr learned my addr 172.21.15.132:0/2886638903 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:27.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.771+0000 7f2ee2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:52334/0 (socket says 172.21.15.132:52334) 2024-03-27T06:16:27.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 -- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 msgr2=0x7f2eec155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:27.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 -- 172.21.15.132:0/2886638903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2eec151830 msgr2=0x7f2eec155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2eec151830 0x7f2eec155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 -- 172.21.15.132:0/2886638903 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec158db0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2ee2a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2eeb400640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2edc02c000 tx=0x7f2edc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc047070 con 0x7f2eec0c64d0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc0378f0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec15b6d0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.772+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc037ce0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.773+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec158ad0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.774+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec0686c0 con 0x7f2eec0c64d0 2024-03-27T06:16:27.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.775+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2edc04a070 con 0x7f2eec0c64d0 2024-03-27T06:16:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.775+0000 7f2ee3e00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 0x7f2ec806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.775+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7f2edc0ad110 con 0x7f2eec0c64d0 2024-03-27T06:16:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.776+0000 7f2ee2a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 0x7f2ec806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:27.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.776+0000 7f2ee2a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 0x7f2ec806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0c2e30 tx=0x7f2ed4007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:27.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:27.780+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2eec158db0 con 0x7f2eec0c64d0 2024-03-27T06:16:28.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.019+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eec070eb0 con 0x7f2eec0c64d0 2024-03-27T06:16:28.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.020+0000 7f2ee3e00640 1 -- 172.21.15.132:0/2886638903 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f2edc033030 con 0x7f2eec0c64d0 2024-03-27T06:16:28.010 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:28.010 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:28.011 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:28.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 msgr2=0x7f2ec806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:28.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 0x7f2ec806e9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0c2e30 tx=0x7f2ed4007d30 comp rx=0 tx=0).stop 2024-03-27T06:16:28.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 msgr2=0x7f2eec154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:28.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2edc02c000 tx=0x7f2edc004730 comp rx=0 tx=0).stop 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 shutdown_connections 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2eec151830 0x7f2eec155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2ec806c530 0x7f2ec806e9f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2eec0c68f0 0x7f2eec155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 --2- 172.21.15.132:0/2886638903 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2eec0c64d0 0x7f2eec154790 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 >> 172.21.15.132:0/2886638903 conn(0x7f2eec0cc7a0 msgr2=0x7f2eec0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 shutdown_connections 2024-03-27T06:16:28.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:28.022+0000 7f2ef0a00640 1 -- 172.21.15.132:0/2886638903 wait complete. 2024-03-27T06:16:28.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:30.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:30.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.265+0000 7f87e4c00640 1 Processor -- start 2024-03-27T06:16:30.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.265+0000 7f87e4c00640 1 -- start start 2024-03-27T06:16:30.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.266+0000 7f87e4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 0x7f87e006b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:30.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.266+0000 7f87e4c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e015b790 con 0x7f87e015aa00 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.266+0000 7f87e4c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e00c12d0 con 0x7f87e00b11e0 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.266+0000 7f87e4c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f87e00c1080 con 0x7f87e006b520 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87d5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 0x7f87e006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87dde00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f87e00b11e0 0x7f87e00c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:36804/0 (socket says 172.21.15.132:36804) 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87dde00640 1 -- 172.21.15.132:0/3421994950 learned_addr learned my addr 172.21.15.132:0/3421994950 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87d5400640 1 -- 172.21.15.132:0/3421994950 >> v1:172.21.15.134:6789/0 conn(0x7f87e00b11e0 legacy=0x7f87e00c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87dd400640 1 -- 172.21.15.132:0/3421994950 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77972145 0 0) 0x7f87e00c12d0 con 0x7f87e00b11e0 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87d5400640 1 -- 172.21.15.132:0/3421994950 >> v1:172.21.15.132:6789/0 conn(0x7f87e015aa00 legacy=0x7f87e015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.267+0000 7f87d5400640 1 -- 172.21.15.132:0/3421994950 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e00c15b0 con 0x7f87e006b520 2024-03-27T06:16:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.268+0000 7f87d5400640 1 --2- 172.21.15.132:0/3421994950 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 0x7f87e006b960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87c80097d0 tx=0x7f87c802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2f688b883658a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.268+0000 7f87dd400640 1 -- 172.21.15.132:0/3421994950 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8034070 con 0x7f87e006b520 2024-03-27T06:16:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87dd400640 1 -- 172.21.15.132:0/3421994950 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8030890 con 0x7f87e006b520 2024-03-27T06:16:30.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87dd400640 1 -- 172.21.15.132:0/3421994950 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8030c60 con 0x7f87e006b520 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 -- 172.21.15.132:0/3421994950 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 msgr2=0x7f87e006b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3421994950 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 0x7f87e006b960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87c80097d0 tx=0x7f87c802b9d0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 -- 172.21.15.132:0/3421994950 shutdown_connections 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3421994950 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e006b520 0x7f87e006b960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 -- 172.21.15.132:0/3421994950 >> 172.21.15.132:0/3421994950 conn(0x7f87e00cc060 msgr2=0x7f87e00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 -- 172.21.15.132:0/3421994950 shutdown_connections 2024-03-27T06:16:30.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.269+0000 7f87e4c00640 1 -- 172.21.15.132:0/3421994950 wait complete. 2024-03-27T06:16:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.271+0000 7f87e4c00640 1 Processor -- start 2024-03-27T06:16:30.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.271+0000 7f87e4c00640 1 -- start start 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87e006b520 0x7f87e00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e015aa00 0x7f87e01f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87e00b15c0 con 0x7f87e006b520 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87e00c1c00 con 0x7f87e015aa00 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87e4c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f87e00c2090 con 0x7f87e00b11e0 2024-03-27T06:16:30.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87dde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87e006b520 0x7f87e00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87d5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87d5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59038/0 (socket says 172.21.15.132:59038) 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87d5400640 1 -- 172.21.15.132:0/3698193042 learned_addr learned my addr 172.21.15.132:0/3698193042 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.272+0000 7f87de800640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e015aa00 0x7f87e01f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 -- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e015aa00 msgr2=0x7f87e01f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e015aa00 0x7f87e01f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 -- 172.21.15.132:0/3698193042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87e006b520 msgr2=0x7f87e00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87e006b520 0x7f87e00c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 -- 172.21.15.132:0/3698193042 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e0156960 con 0x7f87e00b11e0 2024-03-27T06:16:30.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d5400640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f87c8000c00 tx=0x7f87c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:30.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8034040 con 0x7f87e00b11e0 2024-03-27T06:16:30.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c803aa10 con 0x7f87e00b11e0 2024-03-27T06:16:30.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e01f3a40 con 0x7f87e00b11e0 2024-03-27T06:16:30.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c803ae00 con 0x7f87e00b11e0 2024-03-27T06:16:30.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.273+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e01f3fe0 con 0x7f87e00b11e0 2024-03-27T06:16:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.274+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e0068840 con 0x7f87e00b11e0 2024-03-27T06:16:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.275+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f87c804c060 con 0x7f87e00b11e0 2024-03-27T06:16:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.276+0000 7f87d6a00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 0x7f87bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:30.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.276+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7f87c80ad5f0 con 0x7f87e00b11e0 2024-03-27T06:16:30.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.276+0000 7f87dde00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 0x7f87bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:30.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.277+0000 7f87dde00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 0x7f87bc06e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007f70 tx=0x7f87d0007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:30.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.281+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f87c807b520 con 0x7f87e00b11e0 2024-03-27T06:16:30.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.516+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87e00737e0 con 0x7f87e00b11e0 2024-03-27T06:16:30.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.517+0000 7f87d6a00640 1 -- 172.21.15.132:0/3698193042 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f87c807f820 con 0x7f87e00b11e0 2024-03-27T06:16:30.509 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:30.509 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:30.509 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:30.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 msgr2=0x7f87bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 0x7f87bc06e9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007f70 tx=0x7f87d0007f00 comp rx=0 tx=0).stop 2024-03-27T06:16:30.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 msgr2=0x7f87e00c3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f87c8000c00 tx=0x7f87c80046c0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 shutdown_connections 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f87bc06c530 0x7f87bc06e9f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e015aa00 0x7f87e01f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f87e00b11e0 0x7f87e00c3d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 --2- 172.21.15.132:0/3698193042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87e006b520 0x7f87e00c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 >> 172.21.15.132:0/3698193042 conn(0x7f87e00cc060 msgr2=0x7f87e00ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 shutdown_connections 2024-03-27T06:16:30.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:30.520+0000 7f87e4c00640 1 -- 172.21.15.132:0/3698193042 wait complete. 2024-03-27T06:16:30.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:32.530 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:32.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.759+0000 7f6eac600640 1 Processor -- start 2024-03-27T06:16:32.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.759+0000 7f6eac600640 1 -- start start 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6ea40b11e0 0x7f6ea40c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea406b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f6ea406ba60 0x7f6ea415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea40c1080 con 0x7f6ea406ba60 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea40c1c00 con 0x7f6ea40b11e0 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eac600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea415b790 con 0x7f6ea40c6fa0 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6ea40b11e0 0x7f6ea40c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea406b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6eaa600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f6ea406ba60 0x7f6ea415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6ea40b11e0 0x7f6ea40c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51418/0 (socket says 172.21.15.132:51418) 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9c00640 1 -- 172.21.15.132:0/924940380 learned_addr learned my addr 172.21.15.132:0/924940380 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:32.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9c00640 1 -- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 msgr2=0x7f6ea406b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.761+0000 7f6ea9c00640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea406b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.762+0000 7f6ea9c00640 1 -- 172.21.15.132:0/924940380 >> v2:172.21.15.132:3300/0 conn(0x7f6ea406ba60 msgr2=0x7f6ea415aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.762+0000 7f6ea9c00640 1 --2- 172.21.15.132:0/924940380 >> v2:172.21.15.132:3300/0 conn(0x7f6ea406ba60 0x7f6ea415aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.762+0000 7f6ea9c00640 1 -- 172.21.15.132:0/924940380 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea40c12d0 con 0x7f6ea40b11e0 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.762+0000 7f6ea9200640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea406b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:32.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.763+0000 7f6ea9c00640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c6a60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ea406c7b0 tx=0x7f6e9802b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8269dff7e2fbf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:32.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.763+0000 7f6ea3e00640 1 -- 172.21.15.132:0/924940380 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98034070 con 0x7f6ea40b11e0 2024-03-27T06:16:32.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.763+0000 7f6ea3e00640 1 -- 172.21.15.132:0/924940380 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e98030850 con 0x7f6ea40b11e0 2024-03-27T06:16:32.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.763+0000 7f6ea3e00640 1 -- 172.21.15.132:0/924940380 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98030c20 con 0x7f6ea40b11e0 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 -- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 msgr2=0x7f6ea40c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c6a60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ea406c7b0 tx=0x7f6e9802b990 comp rx=0 tx=0).stop 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 -- 172.21.15.132:0/924940380 shutdown_connections 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 --2- 172.21.15.132:0/924940380 >> v2:172.21.15.132:3300/0 conn(0x7f6ea406ba60 0x7f6ea415aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea406b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 --2- 172.21.15.132:0/924940380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c6a60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 -- 172.21.15.132:0/924940380 >> 172.21.15.132:0/924940380 conn(0x7f6ea40cc060 msgr2=0x7f6ea40cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 -- 172.21.15.132:0/924940380 shutdown_connections 2024-03-27T06:16:32.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.764+0000 7f6eac600640 1 -- 172.21.15.132:0/924940380 wait complete. 2024-03-27T06:16:32.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.766+0000 7f6eac600640 1 Processor -- start 2024-03-27T06:16:32.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.766+0000 7f6eac600640 1 -- start start 2024-03-27T06:16:32.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ea406ba60 0x7f6ea40c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea40b15c0 con 0x7f6ea406ba60 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea40c1890 con 0x7f6ea40b11e0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eac600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea40c2090 con 0x7f6ea40c6fa0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ea406ba60 0x7f6ea40c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eaa600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eaa600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40130/0 (socket says 172.21.15.132:40130) 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eaa600640 1 -- 172.21.15.132:0/87292144 learned_addr learned my addr 172.21.15.132:0/87292144 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51422/0 (socket says 172.21.15.132:51422) 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 -- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 msgr2=0x7f6ea414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 -- 172.21.15.132:0/87292144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ea406ba60 msgr2=0x7f6ea40c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ea406ba60 0x7f6ea40c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6ea9200640 1 -- 172.21.15.132:0/87292144 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea40c3400 con 0x7f6ea40b11e0 2024-03-27T06:16:32.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.768+0000 7f6eaa600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6ea9200640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000e870 tx=0x7f6e9000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90017070 con 0x7f6ea40b11e0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9000c940 con 0x7f6ea40b11e0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9000cd10 con 0x7f6ea40b11e0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea40c36e0 con 0x7f6ea40b11e0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.769+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea4154d20 con 0x7f6ea40b11e0 2024-03-27T06:16:32.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.770+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea4068840 con 0x7f6ea40b11e0 2024-03-27T06:16:32.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.771+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6e9001a070 con 0x7f6ea40b11e0 2024-03-27T06:16:32.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.772+0000 7f6ea1600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 0x7f6e8406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:32.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.772+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7f6e9008b870 con 0x7f6ea40b11e0 2024-03-27T06:16:32.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.772+0000 7f6ea9c00640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 0x7f6e8406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:32.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.773+0000 7f6ea9c00640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 0x7f6e8406e9f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6e98036000 tx=0x7f6e98008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:32.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:32.781+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f6e900597a0 con 0x7f6ea40b11e0 2024-03-27T06:16:33.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.017+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ea406c570 con 0x7f6ea40b11e0 2024-03-27T06:16:33.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.019+0000 7f6ea1600640 1 -- 172.21.15.132:0/87292144 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f6e9005daa0 con 0x7f6ea40b11e0 2024-03-27T06:16:33.009 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:33.010 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:33.010 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 msgr2=0x7f6e8406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 0x7f6e8406e9f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6e98036000 tx=0x7f6e98008030 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 msgr2=0x7f6ea40c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000e870 tx=0x7f6e9000ed30 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 shutdown_connections 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6ea40c6fa0 0x7f6ea414a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6e8406c530 0x7f6e8406e9f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea40b11e0 0x7f6ea40c5710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 --2- 172.21.15.132:0/87292144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ea406ba60 0x7f6ea40c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.020+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 >> 172.21.15.132:0/87292144 conn(0x7f6ea40cc060 msgr2=0x7f6ea40cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:33.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.021+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 shutdown_connections 2024-03-27T06:16:33.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:33.021+0000 7f6eac600640 1 -- 172.21.15.132:0/87292144 wait complete. 2024-03-27T06:16:33.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:33.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:16:33.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:16:35.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:35.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.261+0000 7fa83ce00640 1 Processor -- start 2024-03-27T06:16:35.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.261+0000 7fa83ce00640 1 -- start start 2024-03-27T06:16:35.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa83ce00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa838152950 0x7fa8381590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa83ce00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa83ce00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83806b630 con 0x7fa838073970 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa83ce00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa838152d30 con 0x7fa838152950 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa83ce00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa83806b8d0 con 0x7fa838073550 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa835400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa835e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa838152950 0x7fa8381590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa835400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40140/0 (socket says 172.21.15.132:40140) 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa835400640 1 -- 172.21.15.132:0/3318831845 learned_addr learned my addr 172.21.15.132:0/3318831845 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:35.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.262+0000 7fa836800640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa838073970 0x7fa83806cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37138/0 (socket says 172.21.15.132:37138) 2024-03-27T06:16:35.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.263+0000 7fa835400640 1 -- 172.21.15.132:0/3318831845 >> v2:172.21.15.132:3300/0 conn(0x7fa838152950 msgr2=0x7fa8381590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:35.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.263+0000 7fa835400640 1 --2- 172.21.15.132:0/3318831845 >> v2:172.21.15.132:3300/0 conn(0x7fa838152950 0x7fa8381590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.263+0000 7fa835400640 1 -- 172.21.15.132:0/3318831845 >> v1:172.21.15.134:6789/0 conn(0x7fa838073970 legacy=0x7fa83806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.263+0000 7fa835400640 1 -- 172.21.15.132:0/3318831845 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83806bbb0 con 0x7fa838073550 2024-03-27T06:16:35.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.263+0000 7fa834a00640 1 -- 172.21.15.132:0/3318831845 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293764130 0 0) 0x7fa83806b630 con 0x7fa838073970 2024-03-27T06:16:35.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.264+0000 7fa835400640 1 --2- 172.21.15.132:0/3318831845 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa82402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b519e9f09c48ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:35.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.264+0000 7fa834a00640 1 -- 172.21.15.132:0/3318831845 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824034070 con 0x7fa838073550 2024-03-27T06:16:35.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.264+0000 7fa834a00640 1 -- 172.21.15.132:0/3318831845 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa824030840 con 0x7fa838073550 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.264+0000 7fa834a00640 1 -- 172.21.15.132:0/3318831845 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824030c10 con 0x7fa838073550 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 -- 172.21.15.132:0/3318831845 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 msgr2=0x7fa838071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3318831845 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa82402b980 comp rx=0 tx=0).stop 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 -- 172.21.15.132:0/3318831845 shutdown_connections 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3318831845 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa838071170 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3318831845 >> v2:172.21.15.132:3300/0 conn(0x7fa838152950 0x7fa8381590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 -- 172.21.15.132:0/3318831845 >> 172.21.15.132:0/3318831845 conn(0x7fa8380cc060 msgr2=0x7fa8380cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 -- 172.21.15.132:0/3318831845 shutdown_connections 2024-03-27T06:16:35.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.265+0000 7fa83ce00640 1 -- 172.21.15.132:0/3318831845 wait complete. 2024-03-27T06:16:35.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.267+0000 7fa83ce00640 1 Processor -- start 2024-03-27T06:16:35.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.267+0000 7fa83ce00640 1 -- start start 2024-03-27T06:16:35.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 0x7fa83814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa838152950 0x7fa838150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa83806c450 con 0x7fa838152950 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa83806c170 con 0x7fa838073970 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa83ce00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa83806d660 con 0x7fa838073550 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 0x7fa83814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 0x7fa83814ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51434/0 (socket says 172.21.15.132:51434) 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835400640 1 -- 172.21.15.132:0/3285521750 learned_addr learned my addr 172.21.15.132:0/3285521750 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40154/0 (socket says 172.21.15.132:40154) 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa836800640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa838152950 0x7fa838150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 -- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 msgr2=0x7fa83814ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 0x7fa83814ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 -- 172.21.15.132:0/3285521750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa838152950 msgr2=0x7fa838150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa838152950 0x7fa838150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.268+0000 7fa835e00640 1 -- 172.21.15.132:0/3285521750 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8381e1bf0 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa835e00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa82800ebf0 tx=0x7fa82800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828018020 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82800cda0 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8280146c0 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8381e1ed0 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.269+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8381e45b0 con 0x7fa838073550 2024-03-27T06:16:35.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.270+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa838069560 con 0x7fa838073550 2024-03-27T06:16:35.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.271+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa828005d10 con 0x7fa838073550 2024-03-27T06:16:35.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.272+0000 7fa82e000640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 0x7fa81806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:35.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.272+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7fa82808c3a0 con 0x7fa838073550 2024-03-27T06:16:35.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.272+0000 7fa835400640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 0x7fa81806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:35.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.273+0000 7fa835400640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 0x7fa81806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa824008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:35.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.278+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa838069560 con 0x7fa838073550 2024-03-27T06:16:35.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.519+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa838068840 con 0x7fa838073550 2024-03-27T06:16:35.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.520+0000 7fa82e000640 1 -- 172.21.15.132:0/3285521750 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7fa828022030 con 0x7fa838073550 2024-03-27T06:16:35.511 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:35.511 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:35.511 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 msgr2=0x7fa81806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 0x7fa81806e9f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa824008030 comp rx=0 tx=0).stop 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 msgr2=0x7fa8381eebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa82800ebf0 tx=0x7fa82800c490 comp rx=0 tx=0).stop 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 shutdown_connections 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa838152950 0x7fa838150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa81806c530 0x7fa81806e9f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa838073970 0x7fa83814ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.523+0000 7fa83ce00640 1 --2- 172.21.15.132:0/3285521750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa838073550 0x7fa8381eebe0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:35.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.524+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 >> 172.21.15.132:0/3285521750 conn(0x7fa8380cc060 msgr2=0x7fa8380cb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:35.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.524+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 shutdown_connections 2024-03-27T06:16:35.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:35.524+0000 7fa83ce00640 1 -- 172.21.15.132:0/3285521750 wait complete. 2024-03-27T06:16:35.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:37.534 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:37.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.764+0000 7f3cf3400640 1 Processor -- start 2024-03-27T06:16:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.764+0000 7f3cf3400640 1 -- start start 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3cec152950 0x7f3cec1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3cec073970 0x7f3cec0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f3cec152d30 con 0x7f3cec073970 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3cec06b630 con 0x7f3cec152950 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf3400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec06b8d0 con 0x7f3cec073550 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cebe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf0a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3cec152950 0x7f3cec1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf1400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f3cec073970 0x7f3cec0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cebe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40158/0 (socket says 172.21.15.132:40158) 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cebe00640 1 -- 172.21.15.132:0/2487694178 learned_addr learned my addr 172.21.15.132:0/2487694178 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf0a00640 1 -- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 msgr2=0x7f3cec071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf0a00640 1 -- 172.21.15.132:0/2487694178 >> v2:172.21.15.132:3300/0 conn(0x7f3cec073970 msgr2=0x7f3cec0716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.766+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/2487694178 >> v2:172.21.15.132:3300/0 conn(0x7f3cec073970 0x7f3cec0716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.767+0000 7f3cf0a00640 1 -- 172.21.15.132:0/2487694178 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec06bbb0 con 0x7f3cec152950 2024-03-27T06:16:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.767+0000 7f3cebe00640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.768+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1590a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0c32b0 tx=0x7f3ce002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a062a690d80221 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.768+0000 7f3ceb400640 1 -- 172.21.15.132:0/2487694178 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00315a0 con 0x7f3cec152950 2024-03-27T06:16:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.768+0000 7f3ceb400640 1 -- 172.21.15.132:0/2487694178 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce0037070 con 0x7f3cec152950 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.768+0000 7f3ceb400640 1 -- 172.21.15.132:0/2487694178 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0031960 con 0x7f3cec152950 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 -- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 msgr2=0x7f3cec1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1590a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0c32b0 tx=0x7f3ce002b840 comp rx=0 tx=0).stop 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 -- 172.21.15.132:0/2487694178 shutdown_connections 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 --2- 172.21.15.132:0/2487694178 >> v2:172.21.15.132:3300/0 conn(0x7f3cec073970 0x7f3cec0716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 --2- 172.21.15.132:0/2487694178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1590a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 -- 172.21.15.132:0/2487694178 >> 172.21.15.132:0/2487694178 conn(0x7f3cec0cc060 msgr2=0x7f3cec0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 -- 172.21.15.132:0/2487694178 shutdown_connections 2024-03-27T06:16:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.769+0000 7f3cf3400640 1 -- 172.21.15.132:0/2487694178 wait complete. 2024-03-27T06:16:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.771+0000 7f3cf3400640 1 Processor -- start 2024-03-27T06:16:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.771+0000 7f3cf3400640 1 -- start start 2024-03-27T06:16:37.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cec073970 0x7f3cec0afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1e30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec06c450 con 0x7f3cec073970 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec06c170 con 0x7f3cec152950 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf3400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec06cf50 con 0x7f3cec073550 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1e30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40174/0 (socket says 172.21.15.132:40174) 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 -- 172.21.15.132:0/3867002133 learned_addr learned my addr 172.21.15.132:0/3867002133 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cebe00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cec073970 0x7f3cec0afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 -- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 msgr2=0x7f3cec1e30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1e30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 -- 172.21.15.132:0/3867002133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cec073970 msgr2=0x7f3cec0afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cec073970 0x7f3cec0afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf0a00640 1 -- 172.21.15.132:0/3867002133 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec1e1c10 con 0x7f3cec073550 2024-03-27T06:16:37.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.773+0000 7f3cf1400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1e30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3cf0a00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009890 tx=0x7f3ce00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0032630 con 0x7f3cec073550 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce00327f0 con 0x7f3cec073550 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cec1e1ef0 con 0x7f3cec073550 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0032be0 con 0x7f3cec073550 2024-03-27T06:16:37.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.774+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cec1e28e0 con 0x7f3cec073550 2024-03-27T06:16:37.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.775+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cec068840 con 0x7f3cec073550 2024-03-27T06:16:37.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.776+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3ce004c020 con 0x7f3cec073550 2024-03-27T06:16:37.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.777+0000 7f3ce8c00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:37.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.777+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5852+0+0 (secure 0 0 0) 0x7f3ce002f020 con 0x7f3cec073550 2024-03-27T06:16:37.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.777+0000 7f3cebe00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:37.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.777+0000 7f3cebe00640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3cec15cb70 tx=0x7f3cd8009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:37.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:37.784+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3ce0040070 con 0x7f3cec073550 2024-03-27T06:16:38.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.022+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cec06e100 con 0x7f3cec073550 2024-03-27T06:16:38.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.023+0000 7f3ce8c00640 1 -- 172.21.15.132:0/3867002133 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6886 (secure 0 0 0) 0x7f3ce007b120 con 0x7f3cec073550 2024-03-27T06:16:38.013 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:38.013 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4718,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:15:39.893111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":30,"addr":"172.21.15.132:6836/3973814804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":3973814804},{"type":"v1","addr":"172.21.15.132:6836","nonce":3973814804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270,4241]},"id":1}]} 2024-03-27T06:16:38.014 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 18 2024-03-27T06:16:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.026+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 msgr2=0x7f3ccc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.026+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3cec15cb70 tx=0x7f3cd8009240 comp rx=0 tx=0).stop 2024-03-27T06:16:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.026+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 msgr2=0x7f3cec1f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.026+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009890 tx=0x7f3ce00046c0 comp rx=0 tx=0).stop 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 shutdown_connections 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f3ccc06c530 0x7f3ccc06e9f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cec152950 0x7f3cec1e30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cec073970 0x7f3cec0afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 --2- 172.21.15.132:0/3867002133 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3cec073550 0x7f3cec1f0d00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 >> 172.21.15.132:0/3867002133 conn(0x7f3cec0cc060 msgr2=0x7f3cec0cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 shutdown_connections 2024-03-27T06:16:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:38.027+0000 7f3cf3400640 1 -- 172.21.15.132:0/3867002133 wait complete. 2024-03-27T06:16:38.037 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-03-27T06:16:40.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:40.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.261+0000 7fd44c000640 1 Processor -- start 2024-03-27T06:16:40.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.261+0000 7fd44c000640 1 -- start start 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.262+0000 7fd44c000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.262+0000 7fd44c000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44406b520 con 0x7fd444151ad0 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.262+0000 7fd44c000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44406b7c0 con 0x7fd444157c80 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.262+0000 7fd44c000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd444072be0 con 0x7fd4441516f0 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd449600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd44a000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd444157c80 0x7fd444161e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37144/0 (socket says 172.21.15.132:37144) 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd44a000640 1 -- 172.21.15.132:0/1161691989 learned_addr learned my addr 172.21.15.132:0/1161691989 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:40.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd449600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40186/0 (socket says 172.21.15.132:40186) 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd448c00640 1 --1- 172.21.15.132:0/1161691989 >> v1:172.21.15.132:6789/0 conn(0x7fd444151ad0 0x7fd444157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37634/0 (socket says 172.21.15.132:37634) 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd449600640 1 -- 172.21.15.132:0/1161691989 >> v1:172.21.15.134:6789/0 conn(0x7fd444157c80 legacy=0x7fd444161e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd443e00640 1 -- 172.21.15.132:0/1161691989 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007476479 0 0) 0x7fd44406b7c0 con 0x7fd444157c80 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd449600640 1 -- 172.21.15.132:0/1161691989 >> v1:172.21.15.132:6789/0 conn(0x7fd444151ad0 legacy=0x7fd444157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.263+0000 7fd449600640 1 -- 172.21.15.132:0/1161691989 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44406baa0 con 0x7fd4441516f0 2024-03-27T06:16:40.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.264+0000 7fd449600640 1 --2- 172.21.15.132:0/1161691989 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd4380097d0 tx=0x7fd43802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7bb05010fcf853 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd443e00640 1 -- 172.21.15.132:0/1161691989 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438034070 con 0x7fd4441516f0 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd443e00640 1 -- 172.21.15.132:0/1161691989 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd438030890 con 0x7fd4441516f0 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd443e00640 1 -- 172.21.15.132:0/1161691989 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438030c60 con 0x7fd4441516f0 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 -- 172.21.15.132:0/1161691989 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 msgr2=0x7fd4440c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 --2- 172.21.15.132:0/1161691989 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd4380097d0 tx=0x7fd43802b9d0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 -- 172.21.15.132:0/1161691989 shutdown_connections 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 --2- 172.21.15.132:0/1161691989 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd4441516f0 0x7fd4440c7e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 -- 172.21.15.132:0/1161691989 >> 172.21.15.132:0/1161691989 conn(0x7fd4440cc060 msgr2=0x7fd4440cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 -- 172.21.15.132:0/1161691989 shutdown_connections 2024-03-27T06:16:40.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.265+0000 7fd44c000640 1 -- 172.21.15.132:0/1161691989 wait complete. 2024-03-27T06:16:40.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.267+0000 7fd44c000640 1 Processor -- start 2024-03-27T06:16:40.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.267+0000 7fd44c000640 1 -- start start 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd444151ad0 0x7fd44414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd44406bd80 con 0x7fd4441516f0 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd444072870 con 0x7fd444151ad0 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44c000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd4440c1080 con 0x7fd444157c80 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd449600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd448c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd444151ad0 0x7fd44414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd449600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32864/0 (socket says 172.21.15.132:32864) 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd449600640 1 -- 172.21.15.132:0/276001805 learned_addr learned my addr 172.21.15.132:0/276001805 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.268+0000 7fd44a000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40202/0 (socket says 172.21.15.132:40202) 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 -- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd444151ad0 msgr2=0x7fd44414ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd444151ad0 0x7fd44414ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 -- 172.21.15.132:0/276001805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 msgr2=0x7fd444166ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 -- 172.21.15.132:0/276001805 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4441566b0 con 0x7fd444157c80 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd449600640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44a000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00c6d0 tx=0x7fd43c00cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c018070 con 0x7fd444157c80 2024-03-27T06:16:40.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd43c004d60 con 0x7fd444157c80 2024-03-27T06:16:40.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd444156950 con 0x7fd444157c80 2024-03-27T06:16:40.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c00a880 con 0x7fd444157c80 2024-03-27T06:16:40.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.269+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd444156f10 con 0x7fd444157c80 2024-03-27T06:16:40.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.270+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd444068820 con 0x7fd444157c80 2024-03-27T06:16:40.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.271+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd43c01b070 con 0x7fd444157c80 2024-03-27T06:16:40.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.272+0000 7fd441600640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 0x7fd42806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.272+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7fd43c08ae00 con 0x7fd444157c80 2024-03-27T06:16:40.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.272+0000 7fd449600640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 0x7fd42806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.273+0000 7fd449600640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 0x7fd42806e9f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd43802ba10 tx=0x7fd43803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.280+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd43c058ce0 con 0x7fd444157c80 2024-03-27T06:16:40.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.518+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd44406dc80 con 0x7fd444157c80 2024-03-27T06:16:40.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.519+0000 7fd441600640 1 -- 172.21.15.132:0/276001805 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6052 (secure 0 0 0) 0x7fd43c05cfe0 con 0x7fd444157c80 2024-03-27T06:16:40.510 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:40.510 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:39.130286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:16:40.510 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 19 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 msgr2=0x7fd42806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 0x7fd42806e9f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd43802ba10 tx=0x7fd43803b040 comp rx=0 tx=0).stop 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 msgr2=0x7fd444155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00c6d0 tx=0x7fd43c00cb90 comp rx=0 tx=0).stop 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 shutdown_connections 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd444157c80 0x7fd444155900 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd42806c530 0x7fd42806e9f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd444151ad0 0x7fd44414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 --2- 172.21.15.132:0/276001805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4441516f0 0x7fd444166ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 >> 172.21.15.132:0/276001805 conn(0x7fd4440cc060 msgr2=0x7fd4440c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:40.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 shutdown_connections 2024-03-27T06:16:40.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.522+0000 7fd44c000640 1 -- 172.21.15.132:0/276001805 wait complete. 2024-03-27T06:16:40.531 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-03-27T06:16:40.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-27T06:16:40.531 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:40.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.748+0000 7fbbe8e00640 1 Processor -- start 2024-03-27T06:16:40.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.748+0000 7fbbe8e00640 1 -- start start 2024-03-27T06:16:40.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe8e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe8e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fbbe4073880 0x7fbbe4157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe8e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe406b520 con 0x7fbbe41516f0 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe8e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fbbe4072be0 con 0x7fbbe4073880 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe8e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe406b7c0 con 0x7fbbe4151ad0 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe2800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fbbe4073880 0x7fbbe4157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe1400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe1400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40204/0 (socket says 172.21.15.132:40204) 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe1e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbbe41516f0 0x7fbbe40c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:37150/0 (socket says 172.21.15.132:37150) 2024-03-27T06:16:40.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.750+0000 7fbbe1400640 1 -- 172.21.15.132:0/1161176951 learned_addr learned my addr 172.21.15.132:0/1161176951 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:40.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.751+0000 7fbbe1400640 1 -- 172.21.15.132:0/1161176951 >> v2:172.21.15.132:3300/0 conn(0x7fbbe4073880 msgr2=0x7fbbe4157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:40.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.751+0000 7fbbe1400640 1 --2- 172.21.15.132:0/1161176951 >> v2:172.21.15.132:3300/0 conn(0x7fbbe4073880 0x7fbbe4157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.751+0000 7fbbe1400640 1 -- 172.21.15.132:0/1161176951 >> v1:172.21.15.134:6789/0 conn(0x7fbbe41516f0 legacy=0x7fbbe40c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.751+0000 7fbbe1400640 1 -- 172.21.15.132:0/1161176951 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe406baa0 con 0x7fbbe4151ad0 2024-03-27T06:16:40.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.751+0000 7fbbe0a00640 1 -- 172.21.15.132:0/1161176951 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300962281 0 0) 0x7fbbe406b520 con 0x7fbbe41516f0 2024-03-27T06:16:40.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.752+0000 7fbbe1400640 1 --2- 172.21.15.132:0/1161176951 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009870 tx=0x7fbbd002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7ef2bc72ea67ef server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.752+0000 7fbbe0a00640 1 -- 172.21.15.132:0/1161176951 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd002f840 con 0x7fbbe4151ad0 2024-03-27T06:16:40.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.752+0000 7fbbe0a00640 1 -- 172.21.15.132:0/1161176951 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd002f9e0 con 0x7fbbe4151ad0 2024-03-27T06:16:40.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.752+0000 7fbbe0a00640 1 -- 172.21.15.132:0/1161176951 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd002fdb0 con 0x7fbbe4151ad0 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 -- 172.21.15.132:0/1161176951 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 msgr2=0x7fbbe4157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/1161176951 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009870 tx=0x7fbbd002ba20 comp rx=0 tx=0).stop 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 -- 172.21.15.132:0/1161176951 shutdown_connections 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/1161176951 >> v2:172.21.15.132:3300/0 conn(0x7fbbe4073880 0x7fbbe4157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/1161176951 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe4157570 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 -- 172.21.15.132:0/1161176951 >> 172.21.15.132:0/1161176951 conn(0x7fbbe40cc060 msgr2=0x7fbbe40cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 -- 172.21.15.132:0/1161176951 shutdown_connections 2024-03-27T06:16:40.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.753+0000 7fbbe8e00640 1 -- 172.21.15.132:0/1161176951 wait complete. 2024-03-27T06:16:40.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.755+0000 7fbbe8e00640 1 Processor -- start 2024-03-27T06:16:40.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.756+0000 7fbbe8e00640 1 -- start start 2024-03-27T06:16:40.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbbe41516f0 0x7fbbe41f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe41f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe406bd80 con 0x7fbbe41516f0 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4072870 con 0x7fbbe4073880 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe8e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe40c1080 con 0x7fbbe4151ad0 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51460/0 (socket says 172.21.15.132:51460) 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbbe41516f0 0x7fbbe41f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 -- 172.21.15.132:0/4072840300 learned_addr learned my addr 172.21.15.132:0/4072840300 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe2800640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe41f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 -- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 msgr2=0x7fbbe41f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe41f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 -- 172.21.15.132:0/4072840300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbbe41516f0 msgr2=0x7fbbe41f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.757+0000 7fbbe1e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbbe41516f0 0x7fbbe41f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbe1e00640 1 -- 172.21.15.132:0/4072840300 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe41583f0 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbe1e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400c8c0 tx=0x7fbbd400cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4018070 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd400a640 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe4158110 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd400aa30 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.758+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe41571a0 con 0x7fbbe4073880 2024-03-27T06:16:40.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.759+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe41512a0 con 0x7fbbe4073880 2024-03-27T06:16:40.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.760+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbbd401b070 con 0x7fbbe4073880 2024-03-27T06:16:40.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.761+0000 7fbbda000640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 0x7fbbc406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:40.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.761+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7fbbd408aef0 con 0x7fbbe4073880 2024-03-27T06:16:40.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.761+0000 7fbbe1400640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 0x7fbbc406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:40.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.762+0000 7fbbe1400640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 0x7fbbc406e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0036000 tx=0x7fbbd0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:40.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:40.766+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fbbe41512a0 con 0x7fbbe4073880 2024-03-27T06:16:40.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.004+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbe4068840 con 0x7fbbe4073880 2024-03-27T06:16:40.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.004+0000 7fbbda000640 1 -- 172.21.15.132:0/4072840300 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6052 (secure 0 0 0) 0x7fbbd4058cc0 con 0x7fbbe4073880 2024-03-27T06:16:40.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:40.995 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:39.130286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:16:40.995 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 19 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 msgr2=0x7fbbc406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 0x7fbbc406e9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0036000 tx=0x7fbbd0008030 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 msgr2=0x7fbbe414d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400c8c0 tx=0x7fbbd400cd80 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 shutdown_connections 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbbe4151ad0 0x7fbbe41f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbbe41516f0 0x7fbbe41f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbbc406c530 0x7fbbc406e9f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 --2- 172.21.15.132:0/4072840300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe4073880 0x7fbbe414d0a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 >> 172.21.15.132:0/4072840300 conn(0x7fbbe40cc060 msgr2=0x7fbbe414f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:40.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 shutdown_connections 2024-03-27T06:16:40.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:41.007+0000 7fbbe8e00640 1 -- 172.21.15.132:0/4072840300 wait complete. 2024-03-27T06:16:41.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4270": { "addr": "172.21.15.134:6838/2702381292", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2702381292, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2702381292, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:16:39.130286+0000", "qdb_cluster": [ 4270, 4273 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4270, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:16:43.018 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:43.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.245+0000 7fecd7200640 1 Processor -- start 2024-03-27T06:16:43.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.245+0000 7fecd7200640 1 -- start start 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 0x7fecd80cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fecd80cb870 0x7fecd806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fecd80cada0 con 0x7fecd80cb870 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fecd815de10 con 0x7fecd806be10 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd7200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fecd815e0b0 con 0x7fecd81549a0 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd6800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 0x7fecd80cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd6800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:46988/0 (socket says 172.21.15.132:46988) 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd6800640 1 -- 172.21.15.132:0/2691745020 learned_addr learned my addr 172.21.15.132:0/2691745020 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5400640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.132:3300/0 conn(0x7fecd80cb870 0x7fecd806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 -- 172.21.15.132:0/2691745020 >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 msgr2=0x7fecd815adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:43.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 -- 172.21.15.132:0/2691745020 >> v2:172.21.15.132:3300/0 conn(0x7fecd80cb870 msgr2=0x7fecd806b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:43.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.132:3300/0 conn(0x7fecd80cb870 0x7fecd806b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd5e00640 1 -- 172.21.15.132:0/2691745020 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd815e390 con 0x7fecd81549a0 2024-03-27T06:16:43.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.247+0000 7fecd6800640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:43.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.248+0000 7fecd5e00640 1 --2- 172.21.15.132:0/2691745020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 0x7fecd80cd6a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fecd80cdf20 tx=0x7fecc802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd7c9ac62960e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:43.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd4a00640 1 -- 172.21.15.132:0/2691745020 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8035020 con 0x7fecd81549a0 2024-03-27T06:16:43.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd4a00640 1 -- 172.21.15.132:0/2691745020 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc8031830 con 0x7fecd81549a0 2024-03-27T06:16:43.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd4a00640 1 -- 172.21.15.132:0/2691745020 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8031c00 con 0x7fecd81549a0 2024-03-27T06:16:43.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 -- 172.21.15.132:0/2691745020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 msgr2=0x7fecd80cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 --2- 172.21.15.132:0/2691745020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 0x7fecd80cd6a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fecd80cdf20 tx=0x7fecc802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 -- 172.21.15.132:0/2691745020 shutdown_connections 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.134:3300/0 conn(0x7fecd806be10 0x7fecd815adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 --2- 172.21.15.132:0/2691745020 >> v2:172.21.15.132:3300/0 conn(0x7fecd80cb870 0x7fecd806b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 --2- 172.21.15.132:0/2691745020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd81549a0 0x7fecd80cd6a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 -- 172.21.15.132:0/2691745020 >> 172.21.15.132:0/2691745020 conn(0x7fecd806d8d0 msgr2=0x7fecd806dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.249+0000 7fecd7200640 1 -- 172.21.15.132:0/2691745020 shutdown_connections 2024-03-27T06:16:43.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.250+0000 7fecd7200640 1 -- 172.21.15.132:0/2691745020 wait complete. 2024-03-27T06:16:43.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.251+0000 7fecd7200640 1 Processor -- start 2024-03-27T06:16:43.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.251+0000 7fecd7200640 1 -- start start 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecd806be10 0x7fecd8153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 0x7fecd80c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fecd815ec30 con 0x7fecd806be10 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fecd815e950 con 0x7fecd81549a0 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.252+0000 7fecd7200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fecd815a820 con 0x7fecd80cb870 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecd806be10 0x7fecd8153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd6800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 0x7fecd80c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd6800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:46992/0 (socket says 172.21.15.132:46992) 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd6800640 1 -- 172.21.15.132:0/1653339775 learned_addr learned my addr 172.21.15.132:0/1653339775 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:43.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 -- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 msgr2=0x7fecd81f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 -- 172.21.15.132:0/1653339775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecd806be10 msgr2=0x7fecd8153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecd806be10 0x7fecd8153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 -- 172.21.15.132:0/1653339775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd80c74b0 con 0x7fecd80cb870 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd6800640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.253+0000 7fecd5400640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 0x7fecd80c81c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fecc400d620 tx=0x7fecc400da20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.254+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40125f0 con 0x7fecd80cb870 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.254+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc4019070 con 0x7fecd80cb870 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.254+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40129b0 con 0x7fecd80cb870 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.254+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd80c7750 con 0x7fecd80cb870 2024-03-27T06:16:43.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.254+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd80c7d10 con 0x7fecd80cb870 2024-03-27T06:16:43.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.255+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd80686c0 con 0x7fecd80cb870 2024-03-27T06:16:43.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.256+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fecc4012b50 con 0x7fecd80cb870 2024-03-27T06:16:43.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.256+0000 7fecce000640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 0x7fecb806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:43.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.257+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7fecc408bb80 con 0x7fecd80cb870 2024-03-27T06:16:43.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.257+0000 7fecd5e00640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 0x7fecb806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:43.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.257+0000 7fecd5e00640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 0x7fecb806e9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fecc8036000 tx=0x7fecc8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:43.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.261+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fecc4059fa0 con 0x7fecd80cb870 2024-03-27T06:16:43.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.496+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecd81f02e0 con 0x7fecd80cb870 2024-03-27T06:16:43.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.497+0000 7fecce000640 1 -- 172.21.15.132:0/1653339775 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6052 (secure 0 0 0) 0x7fecc40598a0 con 0x7fecd80cb870 2024-03-27T06:16:43.488 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:43.488 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:39.130286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:16:43.488 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 19 2024-03-27T06:16:43.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 msgr2=0x7fecb806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:43.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 0x7fecb806e9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fecc8036000 tx=0x7fecc8008030 comp rx=0 tx=0).stop 2024-03-27T06:16:43.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 msgr2=0x7fecd80c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:43.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 0x7fecd80c81c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fecc400d620 tx=0x7fecc400da20 comp rx=0 tx=0).stop 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 shutdown_connections 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fecb806c530 0x7fecb806e9f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd81549a0 0x7fecd81f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fecd80cb870 0x7fecd80c81c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 --2- 172.21.15.132:0/1653339775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecd806be10 0x7fecd8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 >> 172.21.15.132:0/1653339775 conn(0x7fecd806d8d0 msgr2=0x7fecd8158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:43.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 shutdown_connections 2024-03-27T06:16:43.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:43.500+0000 7fecd7200640 1 -- 172.21.15.132:0/1653339775 wait complete. 2024-03-27T06:16:43.509 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:45.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:45.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.738+0000 7ffbb7200640 1 Processor -- start 2024-03-27T06:16:45.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.738+0000 7ffbb7200640 1 -- start start 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb7200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ffbb0151ad0 0x7ffbb0157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb7200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ffbb0073880 0x7ffbb0157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb7200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb006b520 con 0x7ffbb01516f0 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb7200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb0072be0 con 0x7ffbb0151ad0 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb7200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb006b7c0 con 0x7ffbb0073880 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbaf400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ffbb0151ad0 0x7ffbb0157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbaf400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ffbb0151ad0 0x7ffbb0157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39706/0 (socket says 172.21.15.132:39706) 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbaf400640 1 -- 172.21.15.132:0/3277718680 learned_addr learned my addr 172.21.15.132:0/3277718680 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.740+0000 7ffbb5200640 1 --2- 172.21.15.132:0/3277718680 >> v2:172.21.15.134:3300/0 conn(0x7ffbb0073880 0x7ffbb0157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.741+0000 7ffbaf400640 1 -- 172.21.15.132:0/3277718680 >> v2:172.21.15.134:3300/0 conn(0x7ffbb0073880 msgr2=0x7ffbb0157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:45.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.741+0000 7ffbaf400640 1 --2- 172.21.15.132:0/3277718680 >> v2:172.21.15.134:3300/0 conn(0x7ffbb0073880 0x7ffbb0157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.741+0000 7ffbaf400640 1 -- 172.21.15.132:0/3277718680 >> v1:172.21.15.132:6789/0 conn(0x7ffbb01516f0 legacy=0x7ffbb00c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.741+0000 7ffbaf400640 1 -- 172.21.15.132:0/3277718680 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb006baa0 con 0x7ffbb0151ad0 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.742+0000 7ffbaf400640 1 --2- 172.21.15.132:0/3277718680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0151ad0 0x7ffbb0157570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00c1e60 tx=0x7ffb9c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5408ec6edcf1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.742+0000 7ffbaea00640 1 -- 172.21.15.132:0/3277718680 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c02f850 con 0x7ffbb0151ad0 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.742+0000 7ffbaea00640 1 -- 172.21.15.132:0/3277718680 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb9c02f9f0 con 0x7ffbb0151ad0 2024-03-27T06:16:45.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.742+0000 7ffbaea00640 1 -- 172.21.15.132:0/3277718680 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c02fdc0 con 0x7ffbb0151ad0 2024-03-27T06:16:45.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 -- 172.21.15.132:0/3277718680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0151ad0 msgr2=0x7ffbb0157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 --2- 172.21.15.132:0/3277718680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0151ad0 0x7ffbb0157570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00c1e60 tx=0x7ffb9c02ba30 comp rx=0 tx=0).stop 2024-03-27T06:16:45.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 -- 172.21.15.132:0/3277718680 shutdown_connections 2024-03-27T06:16:45.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 --2- 172.21.15.132:0/3277718680 >> v2:172.21.15.134:3300/0 conn(0x7ffbb0073880 0x7ffbb0157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 --2- 172.21.15.132:0/3277718680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0151ad0 0x7ffbb0157570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 -- 172.21.15.132:0/3277718680 >> 172.21.15.132:0/3277718680 conn(0x7ffbb00cc060 msgr2=0x7ffbb00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:45.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 -- 172.21.15.132:0/3277718680 shutdown_connections 2024-03-27T06:16:45.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.743+0000 7ffbb7200640 1 -- 172.21.15.132:0/3277718680 wait complete. 2024-03-27T06:16:45.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.745+0000 7ffbb7200640 1 Processor -- start 2024-03-27T06:16:45.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.745+0000 7ffbb7200640 1 -- start start 2024-03-27T06:16:45.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 0x7ffbb01416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb006bd80 con 0x7ffbb0073880 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb0072870 con 0x7ffbb0151ad0 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb7200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb00c1080 con 0x7ffbb01516f0 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39712/0 (socket says 172.21.15.132:39712) 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbafe00640 1 -- 172.21.15.132:0/1524768775 learned_addr learned my addr 172.21.15.132:0/1524768775 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50646/0 (socket says 172.21.15.132:50646) 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb5200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 0x7ffbb01416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 -- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 msgr2=0x7ffbb01416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 0x7ffbb01416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 -- 172.21.15.132:0/1524768775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 msgr2=0x7ffbb0157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbaf400640 1 -- 172.21.15.132:0/1524768775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb0154930 con 0x7ffbb01516f0 2024-03-27T06:16:45.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbb5200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 0x7ffbb01416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:45.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.747+0000 7ffbafe00640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffbaf400640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c033000 tx=0x7ffb9c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c047070 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb9c037be0 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c042480 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb0157b90 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.748+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb0154ba0 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.749+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb0068840 con 0x7ffbb01516f0 2024-03-27T06:16:45.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.750+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ffb9c02d3e0 con 0x7ffbb01516f0 2024-03-27T06:16:45.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.751+0000 7ffba7e00640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 0x7ffb9006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:45.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.751+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7ffb9c0ad8a0 con 0x7ffbb01516f0 2024-03-27T06:16:45.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.751+0000 7ffbafe00640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 0x7ffb9006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:45.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.752+0000 7ffbafe00640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 0x7ffb9006ea20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffba0001c30 tx=0x7ffba0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:45.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.755+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ffb9c033030 con 0x7ffbb01516f0 2024-03-27T06:16:45.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.994+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbb00c83b0 con 0x7ffbb01516f0 2024-03-27T06:16:45.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.995+0000 7ffba7e00640 1 -- 172.21.15.132:0/1524768775 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6054 (secure 0 0 0) 0x7ffb9c07b700 con 0x7ffbb01516f0 2024-03-27T06:16:45.986 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:45.986 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:44.151334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:resolve","state_seq":15,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:16:45.986 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 20 2024-03-27T06:16:45.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 msgr2=0x7ffb9006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 0x7ffb9006ea20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffba0001c30 tx=0x7ffba0009040 comp rx=0 tx=0).stop 2024-03-27T06:16:45.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 msgr2=0x7ffbb00ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c033000 tx=0x7ffb9c004730 comp rx=0 tx=0).stop 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 shutdown_connections 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ffb9006c560 0x7ffb9006ea20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbb0151ad0 0x7ffbb01416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ffbb01516f0 0x7ffbb00ca0d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 --2- 172.21.15.132:0/1524768775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbb0073880 0x7ffbb0157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 >> 172.21.15.132:0/1524768775 conn(0x7ffbb00cc060 msgr2=0x7ffbb0150970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 shutdown_connections 2024-03-27T06:16:45.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:45.998+0000 7ffbb7200640 1 -- 172.21.15.132:0/1524768775 wait complete. 2024-03-27T06:16:46.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:48.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:48.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.240+0000 7f1664a00640 1 Processor -- start 2024-03-27T06:16:48.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.240+0000 7f1664a00640 1 -- start start 2024-03-27T06:16:48.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1664a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f16601549a0 0x7f16600cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:48.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1664a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f166015de10 con 0x7f166015adb0 2024-03-27T06:16:48.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1664a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f166015e0b0 con 0x7f16600cb870 2024-03-27T06:16:48.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1664a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f16600cada0 con 0x7f16601549a0 2024-03-27T06:16:48.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1657e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f16600cb870 0x7f166006b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34378/0 (socket says 172.21.15.132:34378) 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f1657e00640 1 -- 172.21.15.132:0/3884833897 learned_addr learned my addr 172.21.15.132:0/3884833897 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.242+0000 7f165e800640 1 --2- 172.21.15.132:0/3884833897 >> v2:172.21.15.132:3300/0 conn(0x7f16601549a0 0x7f16600cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411524124 0 0) 0x7f166015e0b0 con 0x7f16600cb870 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1644003520 con 0x7f16600cb870 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165e800640 1 -- 172.21.15.132:0/3884833897 >> v1:172.21.15.134:6789/0 conn(0x7f16600cb870 legacy=0x7f166006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165e800640 1 -- 172.21.15.132:0/3884833897 >> v1:172.21.15.132:6789/0 conn(0x7f166015adb0 legacy=0x7f166015b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1094962952 0 0) 0x7f166015de10 con 0x7f166015adb0 2024-03-27T06:16:48.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.243+0000 7f165e800640 1 -- 172.21.15.132:0/3884833897 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166015e390 con 0x7f16601549a0 2024-03-27T06:16:48.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.244+0000 7f165e800640 1 --2- 172.21.15.132:0/3884833897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16601549a0 0x7f16600cd6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f16500097a0 tx=0x7f165002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85064020a6d4c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:48.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.244+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165002f810 con 0x7f16601549a0 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.244+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165002f9b0 con 0x7f16601549a0 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.244+0000 7f165de00640 1 -- 172.21.15.132:0/3884833897 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165002fd80 con 0x7f16601549a0 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 -- 172.21.15.132:0/3884833897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16601549a0 msgr2=0x7f16600cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 --2- 172.21.15.132:0/3884833897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16601549a0 0x7f16600cd6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f16500097a0 tx=0x7f165002b9f0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 -- 172.21.15.132:0/3884833897 shutdown_connections 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 --2- 172.21.15.132:0/3884833897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16601549a0 0x7f16600cd6a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 -- 172.21.15.132:0/3884833897 >> 172.21.15.132:0/3884833897 conn(0x7f166006d8d0 msgr2=0x7f166006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 -- 172.21.15.132:0/3884833897 shutdown_connections 2024-03-27T06:16:48.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.245+0000 7f1664a00640 1 -- 172.21.15.132:0/3884833897 wait complete. 2024-03-27T06:16:48.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.247+0000 7f1664a00640 1 Processor -- start 2024-03-27T06:16:48.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.247+0000 7f1664a00640 1 -- start start 2024-03-27T06:16:48.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f16601549a0 0x7f1660146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f166015adb0 0x7f1660146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f166015ec30 con 0x7f166015adb0 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f166015e950 con 0x7f16600cb870 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1664a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f166015a820 con 0x7f16601549a0 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f166015adb0 0x7f1660146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:47018/0 (socket says 172.21.15.132:47018) 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 -- 172.21.15.132:0/702627382 learned_addr learned my addr 172.21.15.132:0/702627382 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:48.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f1657e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f16601549a0 0x7f1660146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 -- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f16601549a0 msgr2=0x7f1660146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f16601549a0 0x7f1660146920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 -- 172.21.15.132:0/702627382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f166015adb0 msgr2=0x7f1660146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f166015adb0 0x7f1660146e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.248+0000 7f165e800640 1 -- 172.21.15.132:0/702627382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166014ef10 con 0x7f16600cb870 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f165e800640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16500098c0 tx=0x7f1650004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650047070 con 0x7f16600cb870 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1650037810 con 0x7f16600cb870 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650037be0 con 0x7f16600cb870 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166014f450 con 0x7f16600cb870 2024-03-27T06:16:48.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.249+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1660153940 con 0x7f16600cb870 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.250+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16600cafd0 con 0x7f16600cb870 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.251+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f165002d3a0 con 0x7f16600cb870 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.252+0000 7f1656a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 0x7f163c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.252+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f16500ad6a0 con 0x7f16600cb870 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.252+0000 7f1657e00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 0x7f163c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:48.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.253+0000 7f1657e00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 0x7f163c06ea20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16600b3060 tx=0x7f1648007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:48.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.257+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f1650033030 con 0x7f16600cb870 2024-03-27T06:16:48.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.492+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16600cbf10 con 0x7f16600cb870 2024-03-27T06:16:48.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.493+0000 7f1656a00640 1 -- 172.21.15.132:0/702627382 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6054 (secure 0 0 0) 0x7f165007b590 con 0x7f16600cb870 2024-03-27T06:16:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:48.484 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:44.151334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:resolve","state_seq":15,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:16:48.484 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 20 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 msgr2=0x7f163c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 0x7f163c06ea20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16600b3060 tx=0x7f1648007d30 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 msgr2=0x7f1660148940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16500098c0 tx=0x7f1650004730 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 shutdown_connections 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f166015adb0 0x7f1660146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f16601549a0 0x7f1660146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f163c06c560 0x7f163c06ea20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 --2- 172.21.15.132:0/702627382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16600cb870 0x7f1660148940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:48.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.496+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 >> 172.21.15.132:0/702627382 conn(0x7f166006d8d0 msgr2=0x7f16600d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:48.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.497+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 shutdown_connections 2024-03-27T06:16:48.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:48.497+0000 7f1664a00640 1 -- 172.21.15.132:0/702627382 wait complete. 2024-03-27T06:16:48.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:50.509 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:50.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.738+0000 7f08acc00640 1 Processor -- start 2024-03-27T06:16:50.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.739+0000 7f08acc00640 1 -- start start 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.740+0000 7f08acc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f08a806cf50 0x7f08a806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.740+0000 7f08acc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a806d920 con 0x7f08a81590a0 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.740+0000 7f08acc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a806b630 con 0x7f08a8152950 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.740+0000 7f08acc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f08a8152d30 con 0x7f08a806cf50 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.740+0000 7f08a5e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f08a8152950 0x7f08a8073550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34386/0 (socket says 172.21.15.132:34386) 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a5e00640 1 -- 172.21.15.132:0/3006377626 learned_addr learned my addr 172.21.15.132:0/3006377626 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:50.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a6800640 1 --2- 172.21.15.132:0/3006377626 >> v2:172.21.15.132:3300/0 conn(0x7f08a806cf50 0x7f08a806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2305131867 0 0) 0x7f08a806b630 con 0x7f08a8152950 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f088c003520 con 0x7f08a8152950 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a6800640 1 -- 172.21.15.132:0/3006377626 >> v1:172.21.15.134:6789/0 conn(0x7f08a8152950 legacy=0x7f08a8073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a6800640 1 -- 172.21.15.132:0/3006377626 >> v1:172.21.15.132:6789/0 conn(0x7f08a81590a0 legacy=0x7f08a8071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496622604 0 0) 0x7f08a806d920 con 0x7f08a81590a0 2024-03-27T06:16:50.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.741+0000 7f08a6800640 1 -- 172.21.15.132:0/3006377626 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a806b8d0 con 0x7f08a806cf50 2024-03-27T06:16:50.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.742+0000 7f08a6800640 1 --2- 172.21.15.132:0/3006377626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a806d3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009d90 tx=0x7f08a002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20da426c2f151c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.742+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0039070 con 0x7f08a806cf50 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.742+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a002ccf0 con 0x7f08a806cf50 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08a5400640 1 -- 172.21.15.132:0/3006377626 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0034640 con 0x7f08a806cf50 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 -- 172.21.15.132:0/3006377626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 msgr2=0x7f08a806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 --2- 172.21.15.132:0/3006377626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a806d3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009d90 tx=0x7f08a002bf40 comp rx=0 tx=0).stop 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 -- 172.21.15.132:0/3006377626 shutdown_connections 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 --2- 172.21.15.132:0/3006377626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a806d3e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 -- 172.21.15.132:0/3006377626 >> 172.21.15.132:0/3006377626 conn(0x7f08a80cc060 msgr2=0x7f08a80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 -- 172.21.15.132:0/3006377626 shutdown_connections 2024-03-27T06:16:50.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.743+0000 7f08acc00640 1 -- 172.21.15.132:0/3006377626 wait complete. 2024-03-27T06:16:50.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.745+0000 7f08acc00640 1 Processor -- start 2024-03-27T06:16:50.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.745+0000 7f08acc00640 1 -- start start 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a8152950 0x7f08a80c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08a81590a0 0x7f08a81551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f08a806c170 con 0x7f08a806cf50 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f08a806be90 con 0x7f08a8152950 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08acc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f08a806c450 con 0x7f08a81590a0 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08a5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:50.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08a5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39738/0 (socket says 172.21.15.132:39738) 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08a5e00640 1 -- 172.21.15.132:0/3150043124 learned_addr learned my addr 172.21.15.132:0/3150043124 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f08a6800640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08a81590a0 0x7f08a81551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.746+0000 7f089fe00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a8152950 0x7f08a80c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 -- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08a81590a0 msgr2=0x7f08a81551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08a81590a0 0x7f08a81551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 -- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a8152950 msgr2=0x7f08a80c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a8152950 0x7f08a80c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 -- 172.21.15.132:0/3150043124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a81f30e0 con 0x7f08a806cf50 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08a5e00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f089800c530 tx=0x7f089800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898017070 con 0x7f08a806cf50 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0898005ce0 con 0x7f08a806cf50 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08980072b0 con 0x7f08a806cf50 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.747+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a80c5280 con 0x7f08a806cf50 2024-03-27T06:16:50.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.748+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a80c5560 con 0x7f08a806cf50 2024-03-27T06:16:50.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.749+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a8068840 con 0x7f08a806cf50 2024-03-27T06:16:50.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.750+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f089801a070 con 0x7f08a806cf50 2024-03-27T06:16:50.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.751+0000 7f089e000640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 0x7f088406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:50.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.751+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f089808aad0 con 0x7f08a806cf50 2024-03-27T06:16:50.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.751+0000 7f089fe00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 0x7f088406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:50.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.752+0000 7f089fe00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 0x7f088406e9f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08a80c31e0 tx=0x7f0890009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:50.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.757+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f0898058a80 con 0x7f08a806cf50 2024-03-27T06:16:50.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.992+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a80c6b30 con 0x7f08a806cf50 2024-03-27T06:16:50.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.993+0000 7f089e000640 1 -- 172.21.15.132:0/3150043124 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6056 (secure 0 0 0) 0x7f089805cd80 con 0x7f08a806cf50 2024-03-27T06:16:50.984 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:50.984 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:49.214375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:16:50.984 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 21 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 msgr2=0x7f088406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 0x7f088406e9f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08a80c31e0 tx=0x7f0890009040 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 msgr2=0x7f08a81f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f089800c530 tx=0x7f089800c9f0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 shutdown_connections 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f08a81590a0 0x7f08a81551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f088406c530 0x7f088406e9f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a8152950 0x7f08a80c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 --2- 172.21.15.132:0/3150043124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a806cf50 0x7f08a81f3b80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 >> 172.21.15.132:0/3150043124 conn(0x7f08a80cc060 msgr2=0x7f08a80cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 shutdown_connections 2024-03-27T06:16:50.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:50.996+0000 7f08acc00640 1 -- 172.21.15.132:0/3150043124 wait complete. 2024-03-27T06:16:51.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:53.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:53.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.233+0000 7f5ec3200640 1 Processor -- start 2024-03-27T06:16:53.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.233+0000 7f5ec3200640 1 -- start start 2024-03-27T06:16:53.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.234+0000 7f5ec3200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ebc152950 0x7f5ebc1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.234+0000 7f5ec3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.234+0000 7f5ec3200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ebc06b630 con 0x7f5ebc073550 2024-03-27T06:16:53.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.234+0000 7f5ec3200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5ebc152d30 con 0x7f5ebc152950 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.234+0000 7f5ec3200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5ebc06b8d0 con 0x7f5ebc073970 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ec1200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ebc152950 0x7f5ebc1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5eb3400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f5ebc073550 0x7f5ebc071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:49830/0 (socket says 172.21.15.132:49830) 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5eb3400640 1 -- 172.21.15.132:0/2300583090 learned_addr learned my addr 172.21.15.132:0/2300583090 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ebc152950 0x7f5ebc1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36680/0 (socket says 172.21.15.132:36680) 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ec1200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:52976/0 (socket says 172.21.15.132:52976) 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 -- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 msgr2=0x7f5ebc06cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 -- 172.21.15.132:0/2300583090 >> v1:172.21.15.132:6789/0 conn(0x7f5ebc073550 legacy=0x7f5ebc071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ebbe00640 1 -- 172.21.15.132:0/2300583090 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ebc06bbb0 con 0x7f5ebc152950 2024-03-27T06:16:53.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.235+0000 7f5ec1200640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:53.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.236+0000 7f5ebbe00640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc152950 0x7f5ebc1590a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0097a0 tx=0x7f5eac02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03a6fedd82eed59 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:53.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.237+0000 7f5ebb400640 1 -- 172.21.15.132:0/2300583090 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac035020 con 0x7f5ebc152950 2024-03-27T06:16:53.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.237+0000 7f5ebb400640 1 -- 172.21.15.132:0/2300583090 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eac031810 con 0x7f5ebc152950 2024-03-27T06:16:53.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.237+0000 7f5ebb400640 1 -- 172.21.15.132:0/2300583090 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac031be0 con 0x7f5ebc152950 2024-03-27T06:16:53.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.237+0000 7f5ec3200640 1 -- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc152950 msgr2=0x7f5ebc1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.237+0000 7f5ec3200640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc152950 0x7f5ebc1590a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0097a0 tx=0x7f5eac02b8b0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 -- 172.21.15.132:0/2300583090 shutdown_connections 2024-03-27T06:16:53.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc06cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 --2- 172.21.15.132:0/2300583090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc152950 0x7f5ebc1590a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 -- 172.21.15.132:0/2300583090 >> 172.21.15.132:0/2300583090 conn(0x7f5ebc0cc060 msgr2=0x7f5ebc0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:53.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 -- 172.21.15.132:0/2300583090 shutdown_connections 2024-03-27T06:16:53.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.238+0000 7f5ec3200640 1 -- 172.21.15.132:0/2300583090 wait complete. 2024-03-27T06:16:53.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.239+0000 7f5ec3200640 1 Processor -- start 2024-03-27T06:16:53.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.239+0000 7f5ec3200640 1 -- start start 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ebc152950 0x7f5ebc1f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc06c450 con 0x7f5ebc152950 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ebc06c170 con 0x7f5ebc073550 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.240+0000 7f5ec3200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5ebc071880 con 0x7f5ebc073970 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36688/0 (socket says 172.21.15.132:36688) 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 -- 172.21.15.132:0/1431636886 learned_addr learned my addr 172.21.15.132:0/1431636886 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ec1200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ebc152950 0x7f5ebc1f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5eb3400640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 -- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 msgr2=0x7f5ebc0cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc0cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 -- 172.21.15.132:0/1431636886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ebc152950 msgr2=0x7f5ebc1f3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ebc152950 0x7f5ebc1f3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 -- 172.21.15.132:0/1431636886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ebc1509e0 con 0x7f5ebc073550 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.241+0000 7f5ebbe00640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098c0 tx=0x7f5eac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.242+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac035040 con 0x7f5ebc073550 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.242+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eac039cf0 con 0x7f5ebc073550 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.242+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ebc157570 con 0x7f5ebc073550 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.242+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac006e10 con 0x7f5ebc073550 2024-03-27T06:16:53.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.242+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ebc157b30 con 0x7f5ebc073550 2024-03-27T06:16:53.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.243+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ebc068840 con 0x7f5ebc073550 2024-03-27T06:16:53.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.244+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5eac038020 con 0x7f5ebc073550 2024-03-27T06:16:53.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.245+0000 7f5eb8c00640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 0x7f5e9806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:53.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.245+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f5eac0ad3d0 con 0x7f5ebc073550 2024-03-27T06:16:53.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.245+0000 7f5eb3400640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 0x7f5e9806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:53.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.245+0000 7f5eb3400640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 0x7f5e9806e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc072e40 tx=0x7f5ea4007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:53.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.250+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5eac07b880 con 0x7f5ebc073550 2024-03-27T06:16:53.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.489+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ebc0c6b30 con 0x7f5ebc073550 2024-03-27T06:16:53.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.489+0000 7f5eb8c00640 1 -- 172.21.15.132:0/1431636886 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6056 (secure 0 0 0) 0x7f5eac07b180 con 0x7f5ebc073550 2024-03-27T06:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:49.214375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273]},"id":1}]} 2024-03-27T06:16:53.480 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 21 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 msgr2=0x7f5e9806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 0x7f5e9806e9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc072e40 tx=0x7f5ea4007e80 comp rx=0 tx=0).stop 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 msgr2=0x7f5ebc1504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098c0 tx=0x7f5eac0046c0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 shutdown_connections 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ebc152950 0x7f5ebc1f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5ebc073970 0x7f5ebc0cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5e9806c530 0x7f5e9806e9f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 --2- 172.21.15.132:0/1431636886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ebc073550 0x7f5ebc1504a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:53.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 >> 172.21.15.132:0/1431636886 conn(0x7f5ebc0cc060 msgr2=0x7f5ebc0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:53.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 shutdown_connections 2024-03-27T06:16:53.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:53.492+0000 7f5ec3200640 1 -- 172.21.15.132:0/1431636886 wait complete. 2024-03-27T06:16:53.503 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4270": { "addr": "172.21.15.134:6838/2702381292", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2702381292, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2702381292, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:16:49.214375+0000", "qdb_cluster": [ 4270, 4273 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4270, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:16:55.504 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:55.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.729+0000 7fba92c00640 1 Processor -- start 2024-03-27T06:16:55.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.729+0000 7fba92c00640 1 -- start start 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fba8c152950 0x7fba8c1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 0x7fba8c071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fba8c073970 0x7fba8c0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fba8c152d30 con 0x7fba8c073970 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fba8c06b630 con 0x7fba8c152950 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba92c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fba8c06b8d0 con 0x7fba8c073550 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba8be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fba8c152950 0x7fba8c1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba90c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fba8c073970 0x7fba8c0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba83400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 0x7fba8c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba8be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fba8c152950 0x7fba8c1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36690/0 (socket says 172.21.15.132:36690) 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.731+0000 7fba8be00640 1 -- 172.21.15.132:0/1953241909 learned_addr learned my addr 172.21.15.132:0/1953241909 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba8be00640 1 -- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 msgr2=0x7fba8c071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba8be00640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 0x7fba8c071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba8be00640 1 -- 172.21.15.132:0/1953241909 >> v2:172.21.15.132:3300/0 conn(0x7fba8c073970 msgr2=0x7fba8c0716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba8be00640 1 --2- 172.21.15.132:0/1953241909 >> v2:172.21.15.132:3300/0 conn(0x7fba8c073970 0x7fba8c0716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba8be00640 1 -- 172.21.15.132:0/1953241909 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c06bbb0 con 0x7fba8c152950 2024-03-27T06:16:55.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.732+0000 7fba83400640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 0x7fba8c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.733+0000 7fba8be00640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c152950 0x7fba8c1590a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0c32b0 tx=0x7fba7c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41b6866e658e6fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.733+0000 7fba8b400640 1 -- 172.21.15.132:0/1953241909 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c02ca80 con 0x7fba8c152950 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.733+0000 7fba8b400640 1 -- 172.21.15.132:0/1953241909 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba7c02cc20 con 0x7fba8c152950 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.733+0000 7fba8b400640 1 -- 172.21.15.132:0/1953241909 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0343f0 con 0x7fba8c152950 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 -- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c152950 msgr2=0x7fba8c1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c152950 0x7fba8c1590a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0c32b0 tx=0x7fba7c02bee0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 -- 172.21.15.132:0/1953241909 shutdown_connections 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 --2- 172.21.15.132:0/1953241909 >> v2:172.21.15.132:3300/0 conn(0x7fba8c073970 0x7fba8c0716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c073550 0x7fba8c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 --2- 172.21.15.132:0/1953241909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c152950 0x7fba8c1590a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 -- 172.21.15.132:0/1953241909 >> 172.21.15.132:0/1953241909 conn(0x7fba8c0cc060 msgr2=0x7fba8c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:55.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 -- 172.21.15.132:0/1953241909 shutdown_connections 2024-03-27T06:16:55.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.734+0000 7fba92c00640 1 -- 172.21.15.132:0/1953241909 wait complete. 2024-03-27T06:16:55.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.736+0000 7fba92c00640 1 Processor -- start 2024-03-27T06:16:55.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.736+0000 7fba92c00640 1 -- start start 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba8c073970 0x7fba8c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 0x7fba8c14de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06c450 con 0x7fba8c073970 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06c170 con 0x7fba8c073550 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba92c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fba8c06cf50 con 0x7fba8c152950 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba8be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba90c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 0x7fba8c14de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba8be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36694/0 (socket says 172.21.15.132:36694) 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba8be00640 1 -- 172.21.15.132:0/2843978324 learned_addr learned my addr 172.21.15.132:0/2843978324 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:55.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.737+0000 7fba83400640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba8c073970 0x7fba8c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 -- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 msgr2=0x7fba8c14de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 0x7fba8c14de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 -- 172.21.15.132:0/2843978324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba8c073970 msgr2=0x7fba8c14d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba8c073970 0x7fba8c14d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 -- 172.21.15.132:0/2843978324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c157940 con 0x7fba8c073550 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba90c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 0x7fba8c14de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba8be00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fba7c036040 tx=0x7fba7c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c043020 con 0x7fba8c073550 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba7c03e450 con 0x7fba8c073550 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c157c20 con 0x7fba8c073550 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.738+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c009510 con 0x7fba8c073550 2024-03-27T06:16:55.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.739+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c0c15b0 con 0x7fba8c073550 2024-03-27T06:16:55.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.740+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8c068840 con 0x7fba8c073550 2024-03-27T06:16:55.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.741+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fba7c046020 con 0x7fba8c073550 2024-03-27T06:16:55.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.741+0000 7fba88c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 0x7fba6806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:55.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.742+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7fba7c0ade60 con 0x7fba8c073550 2024-03-27T06:16:55.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.742+0000 7fba83400640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 0x7fba6806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:55.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.742+0000 7fba83400640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 0x7fba6806e9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fba74009560 tx=0x7fba74009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:55.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.746+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fba7c04d030 con 0x7fba8c073550 2024-03-27T06:16:55.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.983+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba8c0ca870 con 0x7fba8c073550 2024-03-27T06:16:55.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.984+0000 7fba88c00640 1 -- 172.21.15.132:0/2843978324 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6053 (secure 0 0 0) 0x7fba7c07bd80 con 0x7fba8c073550 2024-03-27T06:16:55.974 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:55.974 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:54.241959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:16:55.975 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 22 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 msgr2=0x7fba6806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 0x7fba6806e9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fba74009560 tx=0x7fba74009190 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 msgr2=0x7fba8c1e4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fba7c036040 tx=0x7fba7c004a40 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 shutdown_connections 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fba8c152950 0x7fba8c14de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba8c073970 0x7fba8c14d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fba6806c530 0x7fba6806e9f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 --2- 172.21.15.132:0/2843978324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba8c073550 0x7fba8c1e4330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 >> 172.21.15.132:0/2843978324 conn(0x7fba8c0cc060 msgr2=0x7fba8c0cb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:55.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 shutdown_connections 2024-03-27T06:16:55.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:55.986+0000 7fba92c00640 1 -- 172.21.15.132:0/2843978324 wait complete. 2024-03-27T06:16:55.997 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:57.998 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:16:58.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.215+0000 7f8798800640 1 Processor -- start 2024-03-27T06:16:58.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.216+0000 7f8798800640 1 -- start start 2024-03-27T06:16:58.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8798800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8790158f10 0x7f879015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:58.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8798800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87900c1120 con 0x7f879006b520 2024-03-27T06:16:58.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8798800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87900c13e0 con 0x7f879006d3e0 2024-03-27T06:16:58.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8798800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f879015fc20 con 0x7f8790158f10 2024-03-27T06:16:58.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8795e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8790158f10 0x7f879015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:58.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8795400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f879006d3e0 0x7f879006d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:58612/0 (socket says 172.21.15.132:58612) 2024-03-27T06:16:58.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.217+0000 7f8795400640 1 -- 172.21.15.132:0/2700948008 learned_addr learned my addr 172.21.15.132:0/2700948008 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:16:58.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853444987 0 0) 0x7f87900c13e0 con 0x7f879006d3e0 2024-03-27T06:16:58.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8778003520 con 0x7f879006d3e0 2024-03-27T06:16:58.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867355846 0 0) 0x7f87900c1120 con 0x7f879006b520 2024-03-27T06:16:58.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8795e00640 1 -- 172.21.15.132:0/2700948008 >> v1:172.21.15.134:6789/0 conn(0x7f879006d3e0 legacy=0x7f879006d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8795e00640 1 -- 172.21.15.132:0/2700948008 >> v1:172.21.15.132:6789/0 conn(0x7f879006b520 legacy=0x7f879006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.218+0000 7f8795e00640 1 -- 172.21.15.132:0/2700948008 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87900c16c0 con 0x7f8790158f10 2024-03-27T06:16:58.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.219+0000 7f8795e00640 1 --2- 172.21.15.132:0/2700948008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8790158f10 0x7f879015f6e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87840097a0 tx=0x7f878402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c506b13660a13632 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:58.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.219+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784034070 con 0x7f8790158f10 2024-03-27T06:16:58.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.219+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8784030610 con 0x7f8790158f10 2024-03-27T06:16:58.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.219+0000 7f8794a00640 1 -- 172.21.15.132:0/2700948008 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87840309c0 con 0x7f8790158f10 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 -- 172.21.15.132:0/2700948008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8790158f10 msgr2=0x7f879015f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 --2- 172.21.15.132:0/2700948008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8790158f10 0x7f879015f6e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87840097a0 tx=0x7f878402b960 comp rx=0 tx=0).stop 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 -- 172.21.15.132:0/2700948008 shutdown_connections 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 --2- 172.21.15.132:0/2700948008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8790158f10 0x7f879015f6e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 -- 172.21.15.132:0/2700948008 >> 172.21.15.132:0/2700948008 conn(0x7f87900cc060 msgr2=0x7f87900cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 -- 172.21.15.132:0/2700948008 shutdown_connections 2024-03-27T06:16:58.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.220+0000 7f8798800640 1 -- 172.21.15.132:0/2700948008 wait complete. 2024-03-27T06:16:58.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.221+0000 7f8798800640 1 Processor -- start 2024-03-27T06:16:58.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.222+0000 7f8798800640 1 -- start start 2024-03-27T06:16:58.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.222+0000 7f8798800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 0x7f8790153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8798800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 0x7f87901570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8798800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8798800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87900c1f60 con 0x7f879006b520 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8798800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87900c1c80 con 0x7f8790158f10 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8798800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f879006cec0 con 0x7f879006d3e0 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 0x7f87901570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 0x7f8790153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8796800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 0x7f87901570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:53014/0 (socket says 172.21.15.132:53014) 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795400640 1 -- 172.21.15.132:0/2185622579 learned_addr learned my addr 172.21.15.132:0/2185622579 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8796800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36704/0 (socket says 172.21.15.132:36704) 2024-03-27T06:16:58.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 -- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 msgr2=0x7f87901570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 0x7f87901570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 -- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 msgr2=0x7f87901f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8795e00640 1 -- 172.21.15.132:0/2185622579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879015dc00 con 0x7f879006b520 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.223+0000 7f8796800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:16:58.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f8795e00640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 0x7f8790153c10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8784033000 tx=0x7f87840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784034040 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878403aca0 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784044630 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87901f4790 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.224+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87901f87b0 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.225+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8790068840 con 0x7f879006b520 2024-03-27T06:16:58.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.226+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f878404b060 con 0x7f879006b520 2024-03-27T06:16:58.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.227+0000 7f878a000640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 0x7f877406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:16:58.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.227+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f87840ad3b0 con 0x7f879006b520 2024-03-27T06:16:58.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.227+0000 7f8795400640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 0x7f877406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:16:58.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.228+0000 7f8795400640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 0x7f877406e9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8790157d30 tx=0x7f8780009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:16:58.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.232+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f878403f070 con 0x7f879006b520 2024-03-27T06:16:58.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.469+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87900ccec0 con 0x7f879006b520 2024-03-27T06:16:58.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.471+0000 7f878a000640 1 -- 172.21.15.132:0/2185622579 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6053 (secure 0 0 0) 0x7f878407b2a0 con 0x7f879006b520 2024-03-27T06:16:58.462 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:16:58.462 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:54.241959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4270]},"id":1}]} 2024-03-27T06:16:58.462 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 22 2024-03-27T06:16:58.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 msgr2=0x7f877406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 0x7f877406e9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8790157d30 tx=0x7f8780009040 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 msgr2=0x7f8790153c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 0x7f8790153c10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8784033000 tx=0x7f87840046c0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 shutdown_connections 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f877406c530 0x7f877406e9f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8790158f10 0x7f87901f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f879006d3e0 0x7f87901570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 --2- 172.21.15.132:0/2185622579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f879006b520 0x7f8790153c10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 >> 172.21.15.132:0/2185622579 conn(0x7f87900cc060 msgr2=0x7f8790072760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 shutdown_connections 2024-03-27T06:16:58.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:16:58.474+0000 7f8798800640 1 -- 172.21.15.132:0/2185622579 wait complete. 2024-03-27T06:16:58.484 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:16:59.303 INFO:tasks.workunit.client.0.smithi132.stdout:Create dir 200 to 299 ... 2024-03-27T06:16:59.306 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 200 2024-03-27T06:16:59.329 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 201 2024-03-27T06:16:59.349 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 202 2024-03-27T06:16:59.367 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 203 2024-03-27T06:16:59.389 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 204 2024-03-27T06:16:59.411 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 205 2024-03-27T06:16:59.432 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 206 2024-03-27T06:16:59.455 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 207 2024-03-27T06:16:59.482 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 208 2024-03-27T06:16:59.527 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 209 2024-03-27T06:17:00.189 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 210 2024-03-27T06:17:00.213 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 211 2024-03-27T06:17:00.235 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 212 2024-03-27T06:17:00.258 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 213 2024-03-27T06:17:00.280 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 214 2024-03-27T06:17:00.302 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 215 2024-03-27T06:17:00.324 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 216 2024-03-27T06:17:00.345 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 217 2024-03-27T06:17:00.368 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 218 2024-03-27T06:17:00.389 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 219 2024-03-27T06:17:00.413 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 220 2024-03-27T06:17:00.437 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 221 2024-03-27T06:17:00.465 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 222 2024-03-27T06:17:00.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:00.491 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 223 2024-03-27T06:17:00.511 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 224 2024-03-27T06:17:00.541 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 225 2024-03-27T06:17:00.558 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 226 2024-03-27T06:17:00.580 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 227 2024-03-27T06:17:00.603 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 228 2024-03-27T06:17:00.623 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 229 2024-03-27T06:17:00.640 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 230 2024-03-27T06:17:00.664 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 231 2024-03-27T06:17:00.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.705+0000 7f12b5600640 1 Processor -- start 2024-03-27T06:17:00.695 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 232 2024-03-27T06:17:00.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.705+0000 7f12b5600640 1 -- start start 2024-03-27T06:17:00.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.706+0000 7f12b5600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b01516f0 0x7f12b00c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.706+0000 7f12b5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b0157c80 0x7f12b00c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.706+0000 7f12b5600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b006b520 con 0x7f12b0151ad0 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.706+0000 7f12b5600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f12b0072be0 con 0x7f12b01516f0 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.706+0000 7f12b5600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f12b006b7c0 con 0x7f12b0157c80 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b01516f0 0x7f12b00c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12af400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b0157c80 0x7f12b00c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b01516f0 0x7f12b00c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36720/0 (socket says 172.21.15.132:36720) 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12a6000640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f12b0151ad0 0x7f12b0157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:49844/0 (socket says 172.21.15.132:49844) 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 -- 172.21.15.132:0/2212376296 learned_addr learned my addr 172.21.15.132:0/2212376296 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 -- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b0157c80 msgr2=0x7f12b00c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 --2- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b0157c80 0x7f12b00c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 -- 172.21.15.132:0/2212376296 >> v1:172.21.15.132:6789/0 conn(0x7f12b0151ad0 legacy=0x7f12b0157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.707+0000 7f12aea00640 1 -- 172.21.15.132:0/2212376296 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b006baa0 con 0x7f12b01516f0 2024-03-27T06:17:00.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.708+0000 7f12aea00640 1 --2- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b01516f0 0x7f12b00c7e20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009870 tx=0x7f12a002dc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af75f18012b744f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:00.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12ae000640 1 -- 172.21.15.132:0/2212376296 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a003b070 con 0x7f12b01516f0 2024-03-27T06:17:00.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12ae000640 1 -- 172.21.15.132:0/2212376296 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a002e9d0 con 0x7f12b01516f0 2024-03-27T06:17:00.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12ae000640 1 -- 172.21.15.132:0/2212376296 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a002eda0 con 0x7f12b01516f0 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 -- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b01516f0 msgr2=0x7f12b00c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 --2- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b01516f0 0x7f12b00c7e20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009870 tx=0x7f12a002dc20 comp rx=0 tx=0).stop 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 -- 172.21.15.132:0/2212376296 shutdown_connections 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 --2- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b0157c80 0x7f12b00c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 --2- 172.21.15.132:0/2212376296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b01516f0 0x7f12b00c7e20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 -- 172.21.15.132:0/2212376296 >> 172.21.15.132:0/2212376296 conn(0x7f12b00cc060 msgr2=0x7f12b00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.709+0000 7f12b5600640 1 -- 172.21.15.132:0/2212376296 shutdown_connections 2024-03-27T06:17:00.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.710+0000 7f12b5600640 1 -- 172.21.15.132:0/2212376296 wait complete. 2024-03-27T06:17:00.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.711+0000 7f12b5600640 1 Processor -- start 2024-03-27T06:17:00.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.711+0000 7f12b5600640 1 -- start start 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 0x7f12b00cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b0157c80 0x7f12b0149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12b006bd80 con 0x7f12b0157c80 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12b0072870 con 0x7f12b0151ad0 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12b5600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f12b00c1080 con 0x7f12b01516f0 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.712+0000 7f12aea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12a6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 0x7f12b00cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:53036/0 (socket says 172.21.15.132:53036) 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 -- 172.21.15.132:0/858308367 learned_addr learned my addr 172.21.15.132:0/858308367 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12af400640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b0157c80 0x7f12b0149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 -- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 msgr2=0x7f12b00cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 0x7f12b00cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 -- 172.21.15.132:0/858308367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b0157c80 msgr2=0x7f12b0149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b0157c80 0x7f12b0149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 -- 172.21.15.132:0/858308367 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b00c7b10 con 0x7f12b01516f0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12a6000640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 0x7f12b00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12aea00640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12a0000c00 tx=0x7f12a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.713+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0046070 con 0x7f12b01516f0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.714+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a0036d80 con 0x7f12b01516f0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.714+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b014b240 con 0x7f12b01516f0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.714+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a003b070 con 0x7f12b01516f0 2024-03-27T06:17:00.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.714+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b01479d0 con 0x7f12b01516f0 2024-03-27T06:17:00.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.715+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b0068840 con 0x7f12b01516f0 2024-03-27T06:17:00.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.716+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f12a0049070 con 0x7f12b01516f0 2024-03-27T06:17:00.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.717+0000 7f12a7400640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 0x7f128c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:00.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.717+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f12a00acfe0 con 0x7f12b01516f0 2024-03-27T06:17:00.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.717+0000 7f12a6000640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 0x7f128c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:00.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.718+0000 7f12a6000640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 0x7f128c06ea20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12b014a580 tx=0x7f1298007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:00.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.723+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f12a007adf0 con 0x7f12b01516f0 2024-03-27T06:17:00.714 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 233 2024-03-27T06:17:00.733 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 234 2024-03-27T06:17:00.754 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 235 2024-03-27T06:17:00.775 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 236 2024-03-27T06:17:00.792 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 237 2024-03-27T06:17:00.812 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 238 2024-03-27T06:17:00.827 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 239 2024-03-27T06:17:00.843 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 240 2024-03-27T06:17:00.860 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 241 2024-03-27T06:17:00.877 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 242 2024-03-27T06:17:00.894 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 243 2024-03-27T06:17:00.911 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 244 2024-03-27T06:17:00.934 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 245 2024-03-27T06:17:00.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.962+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12b00ca580 con 0x7f12b01516f0 2024-03-27T06:17:00.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.962+0000 7f12a7400640 1 -- 172.21.15.132:0/858308367 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6058 (secure 0 0 0) 0x7f12a007f0f0 con 0x7f12b01516f0 2024-03-27T06:17:00.953 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:00.953 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273,4718]},"id":1}]} 2024-03-27T06:17:00.953 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 23 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 msgr2=0x7f128c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 0x7f128c06ea20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12b014a580 tx=0x7f1298007e80 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 msgr2=0x7f12b01f0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12a0000c00 tx=0x7f12a00046c0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 shutdown_connections 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b0157c80 0x7f12b0149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f128c06c560 0x7f128c06ea20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b0151ad0 0x7f12b00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 --2- 172.21.15.132:0/858308367 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b01516f0 0x7f12b01f0eb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 >> 172.21.15.132:0/858308367 conn(0x7f12b00cc060 msgr2=0x7f12b014f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:00.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.965+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 shutdown_connections 2024-03-27T06:17:00.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:00.966+0000 7f12b5600640 1 -- 172.21.15.132:0/858308367 wait complete. 2024-03-27T06:17:00.961 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 246 2024-03-27T06:17:00.973 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4241 2024-03-27T06:17:00.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2024-03-27T06:17:00.986 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 247 2024-03-27T06:17:01.011 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 248 2024-03-27T06:17:01.037 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 249 2024-03-27T06:17:01.064 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 250 2024-03-27T06:17:01.093 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 251 2024-03-27T06:17:01.388 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:17:01.398+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-27T06:17:03.602 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 252 2024-03-27T06:17:03.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:17:03.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:17:03.630 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 253 2024-03-27T06:17:03.653 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 254 2024-03-27T06:17:03.672 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 255 2024-03-27T06:17:03.695 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 256 2024-03-27T06:17:03.720 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 257 2024-03-27T06:17:03.748 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 258 2024-03-27T06:17:03.776 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 259 2024-03-27T06:17:03.804 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 260 2024-03-27T06:17:03.848 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 261 2024-03-27T06:17:03.855 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 262 2024-03-27T06:17:03.879 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 263 2024-03-27T06:17:03.901 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 264 2024-03-27T06:17:03.923 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 265 2024-03-27T06:17:03.945 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 266 2024-03-27T06:17:03.967 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 267 2024-03-27T06:17:03.991 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 268 2024-03-27T06:17:04.013 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 269 2024-03-27T06:17:04.035 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 270 2024-03-27T06:17:04.058 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 271 2024-03-27T06:17:04.080 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 272 2024-03-27T06:17:04.102 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 273 2024-03-27T06:17:04.124 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 274 2024-03-27T06:17:04.147 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 275 2024-03-27T06:17:04.170 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 276 2024-03-27T06:17:04.193 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 277 2024-03-27T06:17:04.217 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 278 2024-03-27T06:17:04.239 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 279 2024-03-27T06:17:04.262 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 280 2024-03-27T06:17:04.285 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 281 2024-03-27T06:17:04.311 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 282 2024-03-27T06:17:04.337 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 283 2024-03-27T06:17:04.363 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 284 2024-03-27T06:17:04.391 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 285 2024-03-27T06:17:04.416 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 286 2024-03-27T06:17:04.442 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 287 2024-03-27T06:17:04.467 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 288 2024-03-27T06:17:04.493 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 289 2024-03-27T06:17:04.519 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 290 2024-03-27T06:17:04.545 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 291 2024-03-27T06:17:04.564 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 292 2024-03-27T06:17:04.582 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 293 2024-03-27T06:17:04.597 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 294 2024-03-27T06:17:04.612 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 295 2024-03-27T06:17:04.633 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 296 2024-03-27T06:17:04.650 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 297 2024-03-27T06:17:04.672 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 298 2024-03-27T06:17:04.697 INFO:tasks.workunit.client.0.smithi132.stdout: create dir 299 2024-03-27T06:17:04.722 INFO:tasks.workunit.client.0.smithi132.stdout:Create a snapshot in every first level dir ... 2024-03-27T06:17:04.946 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/.snap/snap-subdir-test 2024-03-27T06:17:04.994 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/30 file after the snapshot 2024-03-27T06:17:04.996 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/31 file after the snapshot 2024-03-27T06:17:04.998 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/32 file after the snapshot 2024-03-27T06:17:05.000 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/33 file after the snapshot 2024-03-27T06:17:05.001 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/34 file after the snapshot 2024-03-27T06:17:05.003 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/35 file after the snapshot 2024-03-27T06:17:05.005 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/36 file after the snapshot 2024-03-27T06:17:05.007 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/37 file after the snapshot 2024-03-27T06:17:05.009 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/38 file after the snapshot 2024-03-27T06:17:05.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 100/39 file after the snapshot 2024-03-27T06:17:05.012 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/.snap/snap-subdir-test 2024-03-27T06:17:05.065 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/30 file after the snapshot 2024-03-27T06:17:05.067 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/31 file after the snapshot 2024-03-27T06:17:05.068 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/32 file after the snapshot 2024-03-27T06:17:05.069 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/33 file after the snapshot 2024-03-27T06:17:05.071 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/34 file after the snapshot 2024-03-27T06:17:05.072 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/35 file after the snapshot 2024-03-27T06:17:05.073 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/36 file after the snapshot 2024-03-27T06:17:05.075 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/37 file after the snapshot 2024-03-27T06:17:05.076 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/38 file after the snapshot 2024-03-27T06:17:05.077 INFO:tasks.workunit.client.0.smithi132.stdout: create 101/39 file after the snapshot 2024-03-27T06:17:05.079 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/.snap/snap-subdir-test 2024-03-27T06:17:05.124 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/30 file after the snapshot 2024-03-27T06:17:05.125 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/31 file after the snapshot 2024-03-27T06:17:05.126 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/32 file after the snapshot 2024-03-27T06:17:05.128 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/33 file after the snapshot 2024-03-27T06:17:05.129 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/34 file after the snapshot 2024-03-27T06:17:05.131 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/35 file after the snapshot 2024-03-27T06:17:05.133 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/36 file after the snapshot 2024-03-27T06:17:05.135 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/37 file after the snapshot 2024-03-27T06:17:05.137 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/38 file after the snapshot 2024-03-27T06:17:05.138 INFO:tasks.workunit.client.0.smithi132.stdout: create 102/39 file after the snapshot 2024-03-27T06:17:05.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/.snap/snap-subdir-test 2024-03-27T06:17:05.185 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/30 file after the snapshot 2024-03-27T06:17:05.187 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/31 file after the snapshot 2024-03-27T06:17:05.189 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/32 file after the snapshot 2024-03-27T06:17:05.191 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/33 file after the snapshot 2024-03-27T06:17:05.193 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/34 file after the snapshot 2024-03-27T06:17:05.195 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/35 file after the snapshot 2024-03-27T06:17:05.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/36 file after the snapshot 2024-03-27T06:17:05.198 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/37 file after the snapshot 2024-03-27T06:17:05.200 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/38 file after the snapshot 2024-03-27T06:17:05.201 INFO:tasks.workunit.client.0.smithi132.stdout: create 103/39 file after the snapshot 2024-03-27T06:17:05.202 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/.snap/snap-subdir-test 2024-03-27T06:17:05.247 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/30 file after the snapshot 2024-03-27T06:17:05.248 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/31 file after the snapshot 2024-03-27T06:17:05.249 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/32 file after the snapshot 2024-03-27T06:17:05.251 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/33 file after the snapshot 2024-03-27T06:17:05.253 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/34 file after the snapshot 2024-03-27T06:17:05.254 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/35 file after the snapshot 2024-03-27T06:17:05.256 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/36 file after the snapshot 2024-03-27T06:17:05.257 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/37 file after the snapshot 2024-03-27T06:17:05.258 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/38 file after the snapshot 2024-03-27T06:17:05.260 INFO:tasks.workunit.client.0.smithi132.stdout: create 104/39 file after the snapshot 2024-03-27T06:17:05.261 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/.snap/snap-subdir-test 2024-03-27T06:17:05.308 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/30 file after the snapshot 2024-03-27T06:17:05.309 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/31 file after the snapshot 2024-03-27T06:17:05.310 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/32 file after the snapshot 2024-03-27T06:17:05.311 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/33 file after the snapshot 2024-03-27T06:17:05.312 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/34 file after the snapshot 2024-03-27T06:17:05.314 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/35 file after the snapshot 2024-03-27T06:17:05.316 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/36 file after the snapshot 2024-03-27T06:17:05.318 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/37 file after the snapshot 2024-03-27T06:17:05.320 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/38 file after the snapshot 2024-03-27T06:17:05.321 INFO:tasks.workunit.client.0.smithi132.stdout: create 105/39 file after the snapshot 2024-03-27T06:17:05.323 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/.snap/snap-subdir-test 2024-03-27T06:17:05.371 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/30 file after the snapshot 2024-03-27T06:17:05.373 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/31 file after the snapshot 2024-03-27T06:17:05.374 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/32 file after the snapshot 2024-03-27T06:17:05.375 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/33 file after the snapshot 2024-03-27T06:17:05.376 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/34 file after the snapshot 2024-03-27T06:17:05.377 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/35 file after the snapshot 2024-03-27T06:17:05.379 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/36 file after the snapshot 2024-03-27T06:17:05.380 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/37 file after the snapshot 2024-03-27T06:17:05.381 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/38 file after the snapshot 2024-03-27T06:17:05.382 INFO:tasks.workunit.client.0.smithi132.stdout: create 106/39 file after the snapshot 2024-03-27T06:17:05.384 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/.snap/snap-subdir-test 2024-03-27T06:17:05.431 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/30 file after the snapshot 2024-03-27T06:17:05.432 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/31 file after the snapshot 2024-03-27T06:17:05.434 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/32 file after the snapshot 2024-03-27T06:17:05.435 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/33 file after the snapshot 2024-03-27T06:17:05.437 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/34 file after the snapshot 2024-03-27T06:17:05.439 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/35 file after the snapshot 2024-03-27T06:17:05.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/36 file after the snapshot 2024-03-27T06:17:05.442 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/37 file after the snapshot 2024-03-27T06:17:05.444 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/38 file after the snapshot 2024-03-27T06:17:05.446 INFO:tasks.workunit.client.0.smithi132.stdout: create 107/39 file after the snapshot 2024-03-27T06:17:05.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/.snap/snap-subdir-test 2024-03-27T06:17:05.497 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/30 file after the snapshot 2024-03-27T06:17:05.498 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/31 file after the snapshot 2024-03-27T06:17:05.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/32 file after the snapshot 2024-03-27T06:17:05.500 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/33 file after the snapshot 2024-03-27T06:17:05.502 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/34 file after the snapshot 2024-03-27T06:17:05.503 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/35 file after the snapshot 2024-03-27T06:17:05.504 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/36 file after the snapshot 2024-03-27T06:17:05.506 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/37 file after the snapshot 2024-03-27T06:17:05.507 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/38 file after the snapshot 2024-03-27T06:17:05.508 INFO:tasks.workunit.client.0.smithi132.stdout: create 108/39 file after the snapshot 2024-03-27T06:17:05.510 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/.snap/snap-subdir-test 2024-03-27T06:17:05.557 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/30 file after the snapshot 2024-03-27T06:17:05.559 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/31 file after the snapshot 2024-03-27T06:17:05.560 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/32 file after the snapshot 2024-03-27T06:17:05.561 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/33 file after the snapshot 2024-03-27T06:17:05.563 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/34 file after the snapshot 2024-03-27T06:17:05.564 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/35 file after the snapshot 2024-03-27T06:17:05.565 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/36 file after the snapshot 2024-03-27T06:17:05.567 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/37 file after the snapshot 2024-03-27T06:17:05.568 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/38 file after the snapshot 2024-03-27T06:17:05.570 INFO:tasks.workunit.client.0.smithi132.stdout: create 109/39 file after the snapshot 2024-03-27T06:17:05.572 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/.snap/snap-subdir-test 2024-03-27T06:17:05.608 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/30 file after the snapshot 2024-03-27T06:17:05.609 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/31 file after the snapshot 2024-03-27T06:17:05.610 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/32 file after the snapshot 2024-03-27T06:17:05.612 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/33 file after the snapshot 2024-03-27T06:17:05.613 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/34 file after the snapshot 2024-03-27T06:17:05.615 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/35 file after the snapshot 2024-03-27T06:17:05.616 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/36 file after the snapshot 2024-03-27T06:17:05.617 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/37 file after the snapshot 2024-03-27T06:17:05.619 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/38 file after the snapshot 2024-03-27T06:17:05.620 INFO:tasks.workunit.client.0.smithi132.stdout: create 110/39 file after the snapshot 2024-03-27T06:17:05.621 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/.snap/snap-subdir-test 2024-03-27T06:17:05.652 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/30 file after the snapshot 2024-03-27T06:17:05.653 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/31 file after the snapshot 2024-03-27T06:17:05.655 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/32 file after the snapshot 2024-03-27T06:17:05.656 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/33 file after the snapshot 2024-03-27T06:17:05.657 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/34 file after the snapshot 2024-03-27T06:17:05.658 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/35 file after the snapshot 2024-03-27T06:17:05.659 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/36 file after the snapshot 2024-03-27T06:17:05.661 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/37 file after the snapshot 2024-03-27T06:17:05.662 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/38 file after the snapshot 2024-03-27T06:17:05.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 111/39 file after the snapshot 2024-03-27T06:17:05.665 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/.snap/snap-subdir-test 2024-03-27T06:17:05.704 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/30 file after the snapshot 2024-03-27T06:17:05.704 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/31 file after the snapshot 2024-03-27T06:17:05.706 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/32 file after the snapshot 2024-03-27T06:17:05.707 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/33 file after the snapshot 2024-03-27T06:17:05.708 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/34 file after the snapshot 2024-03-27T06:17:05.709 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/35 file after the snapshot 2024-03-27T06:17:05.711 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/36 file after the snapshot 2024-03-27T06:17:05.712 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/37 file after the snapshot 2024-03-27T06:17:05.713 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/38 file after the snapshot 2024-03-27T06:17:05.715 INFO:tasks.workunit.client.0.smithi132.stdout: create 112/39 file after the snapshot 2024-03-27T06:17:05.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/.snap/snap-subdir-test 2024-03-27T06:17:05.751 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/30 file after the snapshot 2024-03-27T06:17:05.752 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/31 file after the snapshot 2024-03-27T06:17:05.753 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/32 file after the snapshot 2024-03-27T06:17:05.754 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/33 file after the snapshot 2024-03-27T06:17:05.755 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/34 file after the snapshot 2024-03-27T06:17:05.756 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/35 file after the snapshot 2024-03-27T06:17:05.757 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/36 file after the snapshot 2024-03-27T06:17:05.759 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/37 file after the snapshot 2024-03-27T06:17:05.760 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/38 file after the snapshot 2024-03-27T06:17:05.761 INFO:tasks.workunit.client.0.smithi132.stdout: create 113/39 file after the snapshot 2024-03-27T06:17:05.763 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/.snap/snap-subdir-test 2024-03-27T06:17:05.797 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/30 file after the snapshot 2024-03-27T06:17:05.799 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/31 file after the snapshot 2024-03-27T06:17:05.801 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/32 file after the snapshot 2024-03-27T06:17:05.803 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/33 file after the snapshot 2024-03-27T06:17:05.804 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/34 file after the snapshot 2024-03-27T06:17:05.806 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/35 file after the snapshot 2024-03-27T06:17:05.808 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/36 file after the snapshot 2024-03-27T06:17:05.810 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/37 file after the snapshot 2024-03-27T06:17:05.811 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/38 file after the snapshot 2024-03-27T06:17:05.812 INFO:tasks.workunit.client.0.smithi132.stdout: create 114/39 file after the snapshot 2024-03-27T06:17:05.813 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/.snap/snap-subdir-test 2024-03-27T06:17:05.850 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/30 file after the snapshot 2024-03-27T06:17:05.852 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/31 file after the snapshot 2024-03-27T06:17:05.854 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/32 file after the snapshot 2024-03-27T06:17:05.856 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/33 file after the snapshot 2024-03-27T06:17:05.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/34 file after the snapshot 2024-03-27T06:17:05.860 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/35 file after the snapshot 2024-03-27T06:17:05.862 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/36 file after the snapshot 2024-03-27T06:17:05.864 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/37 file after the snapshot 2024-03-27T06:17:05.865 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/38 file after the snapshot 2024-03-27T06:17:05.867 INFO:tasks.workunit.client.0.smithi132.stdout: create 115/39 file after the snapshot 2024-03-27T06:17:05.869 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/.snap/snap-subdir-test 2024-03-27T06:17:05.905 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/30 file after the snapshot 2024-03-27T06:17:05.906 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/31 file after the snapshot 2024-03-27T06:17:05.908 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/32 file after the snapshot 2024-03-27T06:17:05.910 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/33 file after the snapshot 2024-03-27T06:17:05.912 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/34 file after the snapshot 2024-03-27T06:17:05.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/35 file after the snapshot 2024-03-27T06:17:05.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/36 file after the snapshot 2024-03-27T06:17:05.917 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/37 file after the snapshot 2024-03-27T06:17:05.919 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/38 file after the snapshot 2024-03-27T06:17:05.921 INFO:tasks.workunit.client.0.smithi132.stdout: create 116/39 file after the snapshot 2024-03-27T06:17:05.922 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/.snap/snap-subdir-test 2024-03-27T06:17:05.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/30 file after the snapshot 2024-03-27T06:17:05.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/31 file after the snapshot 2024-03-27T06:17:05.960 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/32 file after the snapshot 2024-03-27T06:17:05.961 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/33 file after the snapshot 2024-03-27T06:17:05.962 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/34 file after the snapshot 2024-03-27T06:17:05.964 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/35 file after the snapshot 2024-03-27T06:17:05.965 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/36 file after the snapshot 2024-03-27T06:17:05.966 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/37 file after the snapshot 2024-03-27T06:17:05.968 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/38 file after the snapshot 2024-03-27T06:17:05.969 INFO:tasks.workunit.client.0.smithi132.stdout: create 117/39 file after the snapshot 2024-03-27T06:17:05.970 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/.snap/snap-subdir-test 2024-03-27T06:17:06.006 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/30 file after the snapshot 2024-03-27T06:17:06.007 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/31 file after the snapshot 2024-03-27T06:17:06.009 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/32 file after the snapshot 2024-03-27T06:17:06.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/33 file after the snapshot 2024-03-27T06:17:06.013 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/34 file after the snapshot 2024-03-27T06:17:06.014 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/35 file after the snapshot 2024-03-27T06:17:06.016 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/36 file after the snapshot 2024-03-27T06:17:06.018 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/37 file after the snapshot 2024-03-27T06:17:06.019 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/38 file after the snapshot 2024-03-27T06:17:06.021 INFO:tasks.workunit.client.0.smithi132.stdout: create 118/39 file after the snapshot 2024-03-27T06:17:06.023 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/.snap/snap-subdir-test 2024-03-27T06:17:06.061 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/30 file after the snapshot 2024-03-27T06:17:06.063 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/31 file after the snapshot 2024-03-27T06:17:06.064 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/32 file after the snapshot 2024-03-27T06:17:06.065 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/33 file after the snapshot 2024-03-27T06:17:06.067 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/34 file after the snapshot 2024-03-27T06:17:06.068 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/35 file after the snapshot 2024-03-27T06:17:06.070 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/36 file after the snapshot 2024-03-27T06:17:06.071 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/37 file after the snapshot 2024-03-27T06:17:06.072 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/38 file after the snapshot 2024-03-27T06:17:06.074 INFO:tasks.workunit.client.0.smithi132.stdout: create 119/39 file after the snapshot 2024-03-27T06:17:06.075 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/.snap/snap-subdir-test 2024-03-27T06:17:06.113 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/30 file after the snapshot 2024-03-27T06:17:06.114 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/31 file after the snapshot 2024-03-27T06:17:06.115 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/32 file after the snapshot 2024-03-27T06:17:06.117 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/33 file after the snapshot 2024-03-27T06:17:06.119 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/34 file after the snapshot 2024-03-27T06:17:06.121 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/35 file after the snapshot 2024-03-27T06:17:06.123 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/36 file after the snapshot 2024-03-27T06:17:06.124 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/37 file after the snapshot 2024-03-27T06:17:06.126 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/38 file after the snapshot 2024-03-27T06:17:06.128 INFO:tasks.workunit.client.0.smithi132.stdout: create 120/39 file after the snapshot 2024-03-27T06:17:06.130 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/.snap/snap-subdir-test 2024-03-27T06:17:06.172 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/30 file after the snapshot 2024-03-27T06:17:06.174 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/31 file after the snapshot 2024-03-27T06:17:06.175 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/32 file after the snapshot 2024-03-27T06:17:06.177 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/33 file after the snapshot 2024-03-27T06:17:06.178 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/34 file after the snapshot 2024-03-27T06:17:06.179 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/35 file after the snapshot 2024-03-27T06:17:06.181 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/36 file after the snapshot 2024-03-27T06:17:06.182 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/37 file after the snapshot 2024-03-27T06:17:06.183 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/38 file after the snapshot 2024-03-27T06:17:06.185 INFO:tasks.workunit.client.0.smithi132.stdout: create 121/39 file after the snapshot 2024-03-27T06:17:06.186 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/.snap/snap-subdir-test 2024-03-27T06:17:06.222 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/30 file after the snapshot 2024-03-27T06:17:06.223 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/31 file after the snapshot 2024-03-27T06:17:06.225 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/32 file after the snapshot 2024-03-27T06:17:06.227 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/33 file after the snapshot 2024-03-27T06:17:06.229 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/34 file after the snapshot 2024-03-27T06:17:06.231 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/35 file after the snapshot 2024-03-27T06:17:06.233 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/36 file after the snapshot 2024-03-27T06:17:06.234 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/37 file after the snapshot 2024-03-27T06:17:06.236 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/38 file after the snapshot 2024-03-27T06:17:06.238 INFO:tasks.workunit.client.0.smithi132.stdout: create 122/39 file after the snapshot 2024-03-27T06:17:06.241 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/.snap/snap-subdir-test 2024-03-27T06:17:06.280 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/30 file after the snapshot 2024-03-27T06:17:06.281 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/31 file after the snapshot 2024-03-27T06:17:06.282 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/32 file after the snapshot 2024-03-27T06:17:06.283 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/33 file after the snapshot 2024-03-27T06:17:06.285 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/34 file after the snapshot 2024-03-27T06:17:06.286 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/35 file after the snapshot 2024-03-27T06:17:06.287 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/36 file after the snapshot 2024-03-27T06:17:06.288 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/37 file after the snapshot 2024-03-27T06:17:06.290 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/38 file after the snapshot 2024-03-27T06:17:06.291 INFO:tasks.workunit.client.0.smithi132.stdout: create 123/39 file after the snapshot 2024-03-27T06:17:06.292 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/.snap/snap-subdir-test 2024-03-27T06:17:06.328 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/30 file after the snapshot 2024-03-27T06:17:06.329 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/31 file after the snapshot 2024-03-27T06:17:06.331 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/32 file after the snapshot 2024-03-27T06:17:06.332 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/33 file after the snapshot 2024-03-27T06:17:06.333 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/34 file after the snapshot 2024-03-27T06:17:06.334 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/35 file after the snapshot 2024-03-27T06:17:06.336 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/36 file after the snapshot 2024-03-27T06:17:06.337 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/37 file after the snapshot 2024-03-27T06:17:06.338 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/38 file after the snapshot 2024-03-27T06:17:06.339 INFO:tasks.workunit.client.0.smithi132.stdout: create 124/39 file after the snapshot 2024-03-27T06:17:06.340 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/.snap/snap-subdir-test 2024-03-27T06:17:06.376 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/30 file after the snapshot 2024-03-27T06:17:06.378 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/31 file after the snapshot 2024-03-27T06:17:06.379 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/32 file after the snapshot 2024-03-27T06:17:06.381 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/33 file after the snapshot 2024-03-27T06:17:06.382 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/34 file after the snapshot 2024-03-27T06:17:06.384 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/35 file after the snapshot 2024-03-27T06:17:06.385 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/36 file after the snapshot 2024-03-27T06:17:06.387 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/37 file after the snapshot 2024-03-27T06:17:06.389 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/38 file after the snapshot 2024-03-27T06:17:06.390 INFO:tasks.workunit.client.0.smithi132.stdout: create 125/39 file after the snapshot 2024-03-27T06:17:06.391 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/.snap/snap-subdir-test 2024-03-27T06:17:06.431 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/30 file after the snapshot 2024-03-27T06:17:06.433 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/31 file after the snapshot 2024-03-27T06:17:06.435 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/32 file after the snapshot 2024-03-27T06:17:06.437 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/33 file after the snapshot 2024-03-27T06:17:06.439 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/34 file after the snapshot 2024-03-27T06:17:06.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/35 file after the snapshot 2024-03-27T06:17:06.442 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/36 file after the snapshot 2024-03-27T06:17:06.446 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/37 file after the snapshot 2024-03-27T06:17:06.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/38 file after the snapshot 2024-03-27T06:17:06.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 126/39 file after the snapshot 2024-03-27T06:17:06.449 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/.snap/snap-subdir-test 2024-03-27T06:17:06.490 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/30 file after the snapshot 2024-03-27T06:17:06.491 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/31 file after the snapshot 2024-03-27T06:17:06.492 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/32 file after the snapshot 2024-03-27T06:17:06.494 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/33 file after the snapshot 2024-03-27T06:17:06.496 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/34 file after the snapshot 2024-03-27T06:17:06.498 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/35 file after the snapshot 2024-03-27T06:17:06.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/36 file after the snapshot 2024-03-27T06:17:06.501 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/37 file after the snapshot 2024-03-27T06:17:06.503 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/38 file after the snapshot 2024-03-27T06:17:06.504 INFO:tasks.workunit.client.0.smithi132.stdout: create 127/39 file after the snapshot 2024-03-27T06:17:06.507 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/.snap/snap-subdir-test 2024-03-27T06:17:06.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/30 file after the snapshot 2024-03-27T06:17:06.546 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/31 file after the snapshot 2024-03-27T06:17:06.548 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/32 file after the snapshot 2024-03-27T06:17:06.549 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/33 file after the snapshot 2024-03-27T06:17:06.551 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/34 file after the snapshot 2024-03-27T06:17:06.552 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/35 file after the snapshot 2024-03-27T06:17:06.554 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/36 file after the snapshot 2024-03-27T06:17:06.556 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/37 file after the snapshot 2024-03-27T06:17:06.557 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/38 file after the snapshot 2024-03-27T06:17:06.559 INFO:tasks.workunit.client.0.smithi132.stdout: create 128/39 file after the snapshot 2024-03-27T06:17:06.560 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/.snap/snap-subdir-test 2024-03-27T06:17:06.599 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/30 file after the snapshot 2024-03-27T06:17:06.600 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/31 file after the snapshot 2024-03-27T06:17:06.602 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/32 file after the snapshot 2024-03-27T06:17:06.603 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/33 file after the snapshot 2024-03-27T06:17:06.605 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/34 file after the snapshot 2024-03-27T06:17:06.606 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/35 file after the snapshot 2024-03-27T06:17:06.607 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/36 file after the snapshot 2024-03-27T06:17:06.609 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/37 file after the snapshot 2024-03-27T06:17:06.610 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/38 file after the snapshot 2024-03-27T06:17:06.612 INFO:tasks.workunit.client.0.smithi132.stdout: create 129/39 file after the snapshot 2024-03-27T06:17:06.613 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/.snap/snap-subdir-test 2024-03-27T06:17:06.658 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/30 file after the snapshot 2024-03-27T06:17:06.659 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/31 file after the snapshot 2024-03-27T06:17:06.660 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/32 file after the snapshot 2024-03-27T06:17:06.661 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/33 file after the snapshot 2024-03-27T06:17:06.662 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/34 file after the snapshot 2024-03-27T06:17:06.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/35 file after the snapshot 2024-03-27T06:17:06.664 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/36 file after the snapshot 2024-03-27T06:17:06.666 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/37 file after the snapshot 2024-03-27T06:17:06.667 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/38 file after the snapshot 2024-03-27T06:17:06.668 INFO:tasks.workunit.client.0.smithi132.stdout: create 130/39 file after the snapshot 2024-03-27T06:17:06.669 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/.snap/snap-subdir-test 2024-03-27T06:17:06.710 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/30 file after the snapshot 2024-03-27T06:17:06.711 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/31 file after the snapshot 2024-03-27T06:17:06.712 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/32 file after the snapshot 2024-03-27T06:17:06.714 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/33 file after the snapshot 2024-03-27T06:17:06.715 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/34 file after the snapshot 2024-03-27T06:17:06.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/35 file after the snapshot 2024-03-27T06:17:06.717 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/36 file after the snapshot 2024-03-27T06:17:06.719 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/37 file after the snapshot 2024-03-27T06:17:06.720 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/38 file after the snapshot 2024-03-27T06:17:06.722 INFO:tasks.workunit.client.0.smithi132.stdout: create 131/39 file after the snapshot 2024-03-27T06:17:06.723 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/.snap/snap-subdir-test 2024-03-27T06:17:06.758 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/30 file after the snapshot 2024-03-27T06:17:06.759 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/31 file after the snapshot 2024-03-27T06:17:06.760 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/32 file after the snapshot 2024-03-27T06:17:06.762 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/33 file after the snapshot 2024-03-27T06:17:06.763 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/34 file after the snapshot 2024-03-27T06:17:06.764 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/35 file after the snapshot 2024-03-27T06:17:06.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/36 file after the snapshot 2024-03-27T06:17:06.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/37 file after the snapshot 2024-03-27T06:17:06.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/38 file after the snapshot 2024-03-27T06:17:06.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 132/39 file after the snapshot 2024-03-27T06:17:06.771 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/.snap/snap-subdir-test 2024-03-27T06:17:06.809 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/30 file after the snapshot 2024-03-27T06:17:06.811 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/31 file after the snapshot 2024-03-27T06:17:06.812 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/32 file after the snapshot 2024-03-27T06:17:06.813 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/33 file after the snapshot 2024-03-27T06:17:06.814 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/34 file after the snapshot 2024-03-27T06:17:06.815 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/35 file after the snapshot 2024-03-27T06:17:06.816 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/36 file after the snapshot 2024-03-27T06:17:06.817 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/37 file after the snapshot 2024-03-27T06:17:06.818 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/38 file after the snapshot 2024-03-27T06:17:06.819 INFO:tasks.workunit.client.0.smithi132.stdout: create 133/39 file after the snapshot 2024-03-27T06:17:06.820 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/.snap/snap-subdir-test 2024-03-27T06:17:06.853 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/30 file after the snapshot 2024-03-27T06:17:06.857 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/31 file after the snapshot 2024-03-27T06:17:06.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/32 file after the snapshot 2024-03-27T06:17:06.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/33 file after the snapshot 2024-03-27T06:17:06.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/34 file after the snapshot 2024-03-27T06:17:06.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/35 file after the snapshot 2024-03-27T06:17:06.860 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/36 file after the snapshot 2024-03-27T06:17:06.861 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/37 file after the snapshot 2024-03-27T06:17:06.862 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/38 file after the snapshot 2024-03-27T06:17:06.863 INFO:tasks.workunit.client.0.smithi132.stdout: create 134/39 file after the snapshot 2024-03-27T06:17:06.864 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/.snap/snap-subdir-test 2024-03-27T06:17:06.901 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/30 file after the snapshot 2024-03-27T06:17:06.903 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/31 file after the snapshot 2024-03-27T06:17:06.904 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/32 file after the snapshot 2024-03-27T06:17:06.905 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/33 file after the snapshot 2024-03-27T06:17:06.907 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/34 file after the snapshot 2024-03-27T06:17:06.908 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/35 file after the snapshot 2024-03-27T06:17:06.910 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/36 file after the snapshot 2024-03-27T06:17:06.911 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/37 file after the snapshot 2024-03-27T06:17:06.912 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/38 file after the snapshot 2024-03-27T06:17:06.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 135/39 file after the snapshot 2024-03-27T06:17:06.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/.snap/snap-subdir-test 2024-03-27T06:17:06.951 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/30 file after the snapshot 2024-03-27T06:17:06.952 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/31 file after the snapshot 2024-03-27T06:17:06.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/32 file after the snapshot 2024-03-27T06:17:06.954 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/33 file after the snapshot 2024-03-27T06:17:06.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/34 file after the snapshot 2024-03-27T06:17:06.957 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/35 file after the snapshot 2024-03-27T06:17:06.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/36 file after the snapshot 2024-03-27T06:17:06.959 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/37 file after the snapshot 2024-03-27T06:17:06.960 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/38 file after the snapshot 2024-03-27T06:17:06.961 INFO:tasks.workunit.client.0.smithi132.stdout: create 136/39 file after the snapshot 2024-03-27T06:17:06.962 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/.snap/snap-subdir-test 2024-03-27T06:17:06.974 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-03-27T06:17:06.974 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-27T06:17:06.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-03-27T06:17:06.982 INFO:tasks.ceph.mds.e:Started 2024-03-27T06:17:06.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:07.001 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/30 file after the snapshot 2024-03-27T06:17:07.002 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/31 file after the snapshot 2024-03-27T06:17:07.003 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/32 file after the snapshot 2024-03-27T06:17:07.004 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/33 file after the snapshot 2024-03-27T06:17:07.005 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/34 file after the snapshot 2024-03-27T06:17:07.006 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/35 file after the snapshot 2024-03-27T06:17:07.007 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/36 file after the snapshot 2024-03-27T06:17:07.008 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/37 file after the snapshot 2024-03-27T06:17:07.009 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/38 file after the snapshot 2024-03-27T06:17:07.010 INFO:tasks.workunit.client.0.smithi132.stdout: create 137/39 file after the snapshot 2024-03-27T06:17:07.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/.snap/snap-subdir-test 2024-03-27T06:17:07.046 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/30 file after the snapshot 2024-03-27T06:17:07.047 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/31 file after the snapshot 2024-03-27T06:17:07.050 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/32 file after the snapshot 2024-03-27T06:17:07.051 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/33 file after the snapshot 2024-03-27T06:17:07.053 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/34 file after the snapshot 2024-03-27T06:17:07.054 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/35 file after the snapshot 2024-03-27T06:17:07.056 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/36 file after the snapshot 2024-03-27T06:17:07.057 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/37 file after the snapshot 2024-03-27T06:17:07.058 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/38 file after the snapshot 2024-03-27T06:17:07.060 INFO:tasks.workunit.client.0.smithi132.stdout: create 138/39 file after the snapshot 2024-03-27T06:17:07.061 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/.snap/snap-subdir-test 2024-03-27T06:17:07.092 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/30 file after the snapshot 2024-03-27T06:17:07.094 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/31 file after the snapshot 2024-03-27T06:17:07.095 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/32 file after the snapshot 2024-03-27T06:17:07.097 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/33 file after the snapshot 2024-03-27T06:17:07.098 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/34 file after the snapshot 2024-03-27T06:17:07.099 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/35 file after the snapshot 2024-03-27T06:17:07.101 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/36 file after the snapshot 2024-03-27T06:17:07.102 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/37 file after the snapshot 2024-03-27T06:17:07.103 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/38 file after the snapshot 2024-03-27T06:17:07.105 INFO:tasks.workunit.client.0.smithi132.stdout: create 139/39 file after the snapshot 2024-03-27T06:17:07.106 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/.snap/snap-subdir-test 2024-03-27T06:17:07.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/30 file after the snapshot 2024-03-27T06:17:07.146 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/31 file after the snapshot 2024-03-27T06:17:07.147 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/32 file after the snapshot 2024-03-27T06:17:07.148 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/33 file after the snapshot 2024-03-27T06:17:07.149 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/34 file after the snapshot 2024-03-27T06:17:07.150 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/35 file after the snapshot 2024-03-27T06:17:07.151 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/36 file after the snapshot 2024-03-27T06:17:07.152 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/37 file after the snapshot 2024-03-27T06:17:07.153 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/38 file after the snapshot 2024-03-27T06:17:07.154 INFO:tasks.workunit.client.0.smithi132.stdout: create 140/39 file after the snapshot 2024-03-27T06:17:07.155 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/.snap/snap-subdir-test 2024-03-27T06:17:07.163 INFO:tasks.ceph.mds.e.smithi132.stdout:starting mds.e at 2024-03-27T06:17:07.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.201+0000 7f145c400640 1 Processor -- start 2024-03-27T06:17:07.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.201+0000 7f145c400640 1 -- start start 2024-03-27T06:17:07.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f145c400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f145406c8a0 0x7f14540c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f145c400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f145406cc80 0x7f14540c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f145c400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14540cdad0 con 0x7f14540c77f0 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f145c400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f14540cc210 con 0x7f145406cc80 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f145c400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f145415b690 con 0x7f145406c8a0 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f1459a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f145406c8a0 0x7f14540c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f1459a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f145406c8a0 0x7f14540c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51532/0 (socket says 172.21.15.132:51532) 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f1459a00640 1 -- 172.21.15.132:0/2104453070 learned_addr learned my addr 172.21.15.132:0/2104453070 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.203+0000 7f1459000640 1 --2- 172.21.15.132:0/2104453070 >> v2:172.21.15.132:3300/0 conn(0x7f145406cc80 0x7f14540c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.204+0000 7f1459a00640 1 -- 172.21.15.132:0/2104453070 >> v2:172.21.15.132:3300/0 conn(0x7f145406cc80 msgr2=0x7f14540c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:07.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.204+0000 7f1459a00640 1 --2- 172.21.15.132:0/2104453070 >> v2:172.21.15.132:3300/0 conn(0x7f145406cc80 0x7f14540c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.204+0000 7f1453e00640 1 -- 172.21.15.132:0/2104453070 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012868783 0 0) 0x7f14540cdad0 con 0x7f14540c77f0 2024-03-27T06:17:07.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.204+0000 7f1459a00640 1 -- 172.21.15.132:0/2104453070 >> v1:172.21.15.132:6789/0 conn(0x7f14540c77f0 legacy=0x7f1454154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.204+0000 7f1459a00640 1 -- 172.21.15.132:0/2104453070 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145415b970 con 0x7f145406c8a0 2024-03-27T06:17:07.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.205+0000 7f1459a00640 1 --2- 172.21.15.132:0/2104453070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406c8a0 0x7f14540c8320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f145408db00 tx=0x7f144802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f5df30b6488eeb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:07.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.206+0000 7f1453e00640 1 -- 172.21.15.132:0/2104453070 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448035020 con 0x7f145406c8a0 2024-03-27T06:17:07.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.206+0000 7f1453e00640 1 -- 172.21.15.132:0/2104453070 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1448031870 con 0x7f145406c8a0 2024-03-27T06:17:07.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.206+0000 7f1453e00640 1 -- 172.21.15.132:0/2104453070 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448031c40 con 0x7f145406c8a0 2024-03-27T06:17:07.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/30 file after the snapshot 2024-03-27T06:17:07.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.206+0000 7f145c400640 1 -- 172.21.15.132:0/2104453070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406c8a0 msgr2=0x7f14540c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.206+0000 7f145c400640 1 --2- 172.21.15.132:0/2104453070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406c8a0 0x7f14540c8320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f145408db00 tx=0x7f144802b910 comp rx=0 tx=0).stop 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 -- 172.21.15.132:0/2104453070 shutdown_connections 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 --2- 172.21.15.132:0/2104453070 >> v2:172.21.15.132:3300/0 conn(0x7f145406cc80 0x7f14540c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 --2- 172.21.15.132:0/2104453070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406c8a0 0x7f14540c8320 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 -- 172.21.15.132:0/2104453070 >> 172.21.15.132:0/2104453070 conn(0x7f14540ccbe0 msgr2=0x7f14540ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 -- 172.21.15.132:0/2104453070 shutdown_connections 2024-03-27T06:17:07.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.207+0000 7f145c400640 1 -- 172.21.15.132:0/2104453070 wait complete. 2024-03-27T06:17:07.199 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/31 file after the snapshot 2024-03-27T06:17:07.199 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/32 file after the snapshot 2024-03-27T06:17:07.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.209+0000 7f145c400640 1 Processor -- start 2024-03-27T06:17:07.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.209+0000 7f145c400640 1 -- start start 2024-03-27T06:17:07.200 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/33 file after the snapshot 2024-03-27T06:17:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145406c8a0 0x7f14541f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 0x7f145414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f145406d440 con 0x7f145406c8a0 2024-03-27T06:17:07.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f145406d2c0 con 0x7f145406cc80 2024-03-27T06:17:07.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145c400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f14540cd700 con 0x7f14540c77f0 2024-03-27T06:17:07.202 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/34 file after the snapshot 2024-03-27T06:17:07.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f1459000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f145a400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 0x7f145414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f1459000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51538/0 (socket says 172.21.15.132:51538) 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f1459000640 1 -- 172.21.15.132:0/3200751737 learned_addr learned my addr 172.21.15.132:0/3200751737 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.210+0000 7f1459a00640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145406c8a0 0x7f14541f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 -- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 msgr2=0x7f145414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 0x7f145414d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 -- 172.21.15.132:0/3200751737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145406c8a0 msgr2=0x7f14541f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145406c8a0 0x7f14541f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 -- 172.21.15.132:0/3200751737 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1454151430 con 0x7f145406cc80 2024-03-27T06:17:07.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f145a400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 0x7f145414d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1459000640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f144000e8f0 tx=0x7f144000edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440017020 con 0x7f145406cc80 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1440004240 con 0x7f145406cc80 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144000c730 con 0x7f145406cc80 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14541516d0 con 0x7f145406cc80 2024-03-27T06:17:07.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.211+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1454151c90 con 0x7f145406cc80 2024-03-27T06:17:07.205 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/35 file after the snapshot 2024-03-27T06:17:07.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.213+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1454068840 con 0x7f145406cc80 2024-03-27T06:17:07.206 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/36 file after the snapshot 2024-03-27T06:17:07.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.213+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f144001a020 con 0x7f145406cc80 2024-03-27T06:17:07.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.214+0000 7f1451600640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 0x7f143406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:07.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.214+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7f144008b960 con 0x7f145406cc80 2024-03-27T06:17:07.207 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/37 file after the snapshot 2024-03-27T06:17:07.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.214+0000 7f1459a00640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 0x7f143406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:07.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.215+0000 7f1459a00640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 0x7f143406e9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1448036000 tx=0x7f144803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:07.208 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/38 file after the snapshot 2024-03-27T06:17:07.208 INFO:tasks.workunit.client.0.smithi132.stdout: create 141/39 file after the snapshot 2024-03-27T06:17:07.209 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/.snap/snap-subdir-test 2024-03-27T06:17:07.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.219+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f1440021030 con 0x7f145406cc80 2024-03-27T06:17:07.245 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/30 file after the snapshot 2024-03-27T06:17:07.246 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/31 file after the snapshot 2024-03-27T06:17:07.247 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/32 file after the snapshot 2024-03-27T06:17:07.248 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/33 file after the snapshot 2024-03-27T06:17:07.249 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/34 file after the snapshot 2024-03-27T06:17:07.250 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/35 file after the snapshot 2024-03-27T06:17:07.252 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/36 file after the snapshot 2024-03-27T06:17:07.253 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/37 file after the snapshot 2024-03-27T06:17:07.254 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/38 file after the snapshot 2024-03-27T06:17:07.255 INFO:tasks.workunit.client.0.smithi132.stdout: create 142/39 file after the snapshot 2024-03-27T06:17:07.255 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/.snap/snap-subdir-test 2024-03-27T06:17:07.290 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/30 file after the snapshot 2024-03-27T06:17:07.291 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/31 file after the snapshot 2024-03-27T06:17:07.292 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/32 file after the snapshot 2024-03-27T06:17:07.294 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/33 file after the snapshot 2024-03-27T06:17:07.295 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/34 file after the snapshot 2024-03-27T06:17:07.297 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/35 file after the snapshot 2024-03-27T06:17:07.299 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/36 file after the snapshot 2024-03-27T06:17:07.300 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/37 file after the snapshot 2024-03-27T06:17:07.301 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/38 file after the snapshot 2024-03-27T06:17:07.303 INFO:tasks.workunit.client.0.smithi132.stdout: create 143/39 file after the snapshot 2024-03-27T06:17:07.304 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/.snap/snap-subdir-test 2024-03-27T06:17:07.340 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/30 file after the snapshot 2024-03-27T06:17:07.341 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/31 file after the snapshot 2024-03-27T06:17:07.343 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/32 file after the snapshot 2024-03-27T06:17:07.344 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/33 file after the snapshot 2024-03-27T06:17:07.345 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/34 file after the snapshot 2024-03-27T06:17:07.347 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/35 file after the snapshot 2024-03-27T06:17:07.348 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/36 file after the snapshot 2024-03-27T06:17:07.349 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/37 file after the snapshot 2024-03-27T06:17:07.350 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/38 file after the snapshot 2024-03-27T06:17:07.351 INFO:tasks.workunit.client.0.smithi132.stdout: create 144/39 file after the snapshot 2024-03-27T06:17:07.352 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/.snap/snap-subdir-test 2024-03-27T06:17:07.387 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/30 file after the snapshot 2024-03-27T06:17:07.388 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/31 file after the snapshot 2024-03-27T06:17:07.389 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/32 file after the snapshot 2024-03-27T06:17:07.390 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/33 file after the snapshot 2024-03-27T06:17:07.391 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/34 file after the snapshot 2024-03-27T06:17:07.392 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/35 file after the snapshot 2024-03-27T06:17:07.393 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/36 file after the snapshot 2024-03-27T06:17:07.394 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/37 file after the snapshot 2024-03-27T06:17:07.396 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/38 file after the snapshot 2024-03-27T06:17:07.397 INFO:tasks.workunit.client.0.smithi132.stdout: create 145/39 file after the snapshot 2024-03-27T06:17:07.399 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/.snap/snap-subdir-test 2024-03-27T06:17:07.438 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/30 file after the snapshot 2024-03-27T06:17:07.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/31 file after the snapshot 2024-03-27T06:17:07.441 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/32 file after the snapshot 2024-03-27T06:17:07.443 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/33 file after the snapshot 2024-03-27T06:17:07.444 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/34 file after the snapshot 2024-03-27T06:17:07.446 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/35 file after the snapshot 2024-03-27T06:17:07.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/36 file after the snapshot 2024-03-27T06:17:07.449 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/37 file after the snapshot 2024-03-27T06:17:07.451 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/38 file after the snapshot 2024-03-27T06:17:07.452 INFO:tasks.workunit.client.0.smithi132.stdout: create 146/39 file after the snapshot 2024-03-27T06:17:07.454 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/.snap/snap-subdir-test 2024-03-27T06:17:07.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.489+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f145414fd60 con 0x7f145406cc80 2024-03-27T06:17:07.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.493+0000 7f1451600640 1 -- 172.21.15.132:0/3200751737 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6058 (secure 0 0 0) 0x7f14400597c0 con 0x7f145406cc80 2024-03-27T06:17:07.485 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:07.485 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4270,4273,4718]},"id":1}]} 2024-03-27T06:17:07.485 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 23 2024-03-27T06:17:07.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 msgr2=0x7f143406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 0x7f143406e9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1448036000 tx=0x7f144803c040 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 msgr2=0x7f14541f1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f144000e8f0 tx=0x7f144000edb0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 shutdown_connections 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f14540c77f0 0x7f145414d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f143406c530 0x7f143406e9f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f145406cc80 0x7f14541f1b80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 --2- 172.21.15.132:0/3200751737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f145406c8a0 0x7f14541f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 >> 172.21.15.132:0/3200751737 conn(0x7f14540ccbe0 msgr2=0x7f1454159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 shutdown_connections 2024-03-27T06:17:07.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:07.496+0000 7f145c400640 1 -- 172.21.15.132:0/3200751737 wait complete. 2024-03-27T06:17:07.488 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/30 file after the snapshot 2024-03-27T06:17:07.489 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/31 file after the snapshot 2024-03-27T06:17:07.490 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/32 file after the snapshot 2024-03-27T06:17:07.492 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/33 file after the snapshot 2024-03-27T06:17:07.493 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/34 file after the snapshot 2024-03-27T06:17:07.494 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/35 file after the snapshot 2024-03-27T06:17:07.496 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/36 file after the snapshot 2024-03-27T06:17:07.497 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/37 file after the snapshot 2024-03-27T06:17:07.498 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/38 file after the snapshot 2024-03-27T06:17:07.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 147/39 file after the snapshot 2024-03-27T06:17:07.500 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/.snap/snap-subdir-test 2024-03-27T06:17:07.506 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-03-27T06:17:07.534 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/30 file after the snapshot 2024-03-27T06:17:07.535 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/31 file after the snapshot 2024-03-27T06:17:07.536 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/32 file after the snapshot 2024-03-27T06:17:07.537 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/33 file after the snapshot 2024-03-27T06:17:07.538 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/34 file after the snapshot 2024-03-27T06:17:07.539 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/35 file after the snapshot 2024-03-27T06:17:07.540 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/36 file after the snapshot 2024-03-27T06:17:07.541 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/37 file after the snapshot 2024-03-27T06:17:07.542 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/38 file after the snapshot 2024-03-27T06:17:07.543 INFO:tasks.workunit.client.0.smithi132.stdout: create 148/39 file after the snapshot 2024-03-27T06:17:07.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/.snap/snap-subdir-test 2024-03-27T06:17:07.582 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/30 file after the snapshot 2024-03-27T06:17:07.583 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/31 file after the snapshot 2024-03-27T06:17:07.585 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/32 file after the snapshot 2024-03-27T06:17:07.586 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/33 file after the snapshot 2024-03-27T06:17:07.587 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/34 file after the snapshot 2024-03-27T06:17:07.588 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/35 file after the snapshot 2024-03-27T06:17:07.589 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/36 file after the snapshot 2024-03-27T06:17:07.590 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/37 file after the snapshot 2024-03-27T06:17:07.592 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/38 file after the snapshot 2024-03-27T06:17:07.593 INFO:tasks.workunit.client.0.smithi132.stdout: create 149/39 file after the snapshot 2024-03-27T06:17:07.594 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/.snap/snap-subdir-test 2024-03-27T06:17:07.626 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/30 file after the snapshot 2024-03-27T06:17:07.627 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/31 file after the snapshot 2024-03-27T06:17:07.629 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/32 file after the snapshot 2024-03-27T06:17:07.630 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/33 file after the snapshot 2024-03-27T06:17:07.631 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/34 file after the snapshot 2024-03-27T06:17:07.633 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/35 file after the snapshot 2024-03-27T06:17:07.634 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/36 file after the snapshot 2024-03-27T06:17:07.635 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/37 file after the snapshot 2024-03-27T06:17:07.637 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/38 file after the snapshot 2024-03-27T06:17:07.638 INFO:tasks.workunit.client.0.smithi132.stdout: create 150/39 file after the snapshot 2024-03-27T06:17:07.639 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/.snap/snap-subdir-test 2024-03-27T06:17:07.679 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/30 file after the snapshot 2024-03-27T06:17:07.680 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/31 file after the snapshot 2024-03-27T06:17:07.682 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/32 file after the snapshot 2024-03-27T06:17:07.683 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/33 file after the snapshot 2024-03-27T06:17:07.685 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/34 file after the snapshot 2024-03-27T06:17:07.687 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/35 file after the snapshot 2024-03-27T06:17:07.688 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/36 file after the snapshot 2024-03-27T06:17:07.690 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/37 file after the snapshot 2024-03-27T06:17:07.692 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/38 file after the snapshot 2024-03-27T06:17:07.693 INFO:tasks.workunit.client.0.smithi132.stdout: create 151/39 file after the snapshot 2024-03-27T06:17:07.694 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/.snap/snap-subdir-test 2024-03-27T06:17:07.735 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/30 file after the snapshot 2024-03-27T06:17:07.736 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/31 file after the snapshot 2024-03-27T06:17:07.738 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/32 file after the snapshot 2024-03-27T06:17:07.739 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/33 file after the snapshot 2024-03-27T06:17:07.740 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/34 file after the snapshot 2024-03-27T06:17:07.742 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/35 file after the snapshot 2024-03-27T06:17:07.743 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/36 file after the snapshot 2024-03-27T06:17:07.744 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/37 file after the snapshot 2024-03-27T06:17:07.746 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/38 file after the snapshot 2024-03-27T06:17:07.747 INFO:tasks.workunit.client.0.smithi132.stdout: create 152/39 file after the snapshot 2024-03-27T06:17:07.748 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/.snap/snap-subdir-test 2024-03-27T06:17:07.781 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/30 file after the snapshot 2024-03-27T06:17:07.783 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/31 file after the snapshot 2024-03-27T06:17:07.784 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/32 file after the snapshot 2024-03-27T06:17:07.785 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/33 file after the snapshot 2024-03-27T06:17:07.787 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/34 file after the snapshot 2024-03-27T06:17:07.788 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/35 file after the snapshot 2024-03-27T06:17:07.789 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/36 file after the snapshot 2024-03-27T06:17:07.791 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/37 file after the snapshot 2024-03-27T06:17:07.792 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/38 file after the snapshot 2024-03-27T06:17:07.793 INFO:tasks.workunit.client.0.smithi132.stdout: create 153/39 file after the snapshot 2024-03-27T06:17:07.795 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/.snap/snap-subdir-test 2024-03-27T06:17:07.832 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/30 file after the snapshot 2024-03-27T06:17:07.834 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/31 file after the snapshot 2024-03-27T06:17:07.836 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/32 file after the snapshot 2024-03-27T06:17:07.838 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/33 file after the snapshot 2024-03-27T06:17:07.840 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/34 file after the snapshot 2024-03-27T06:17:07.842 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/35 file after the snapshot 2024-03-27T06:17:07.844 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/36 file after the snapshot 2024-03-27T06:17:07.845 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/37 file after the snapshot 2024-03-27T06:17:07.847 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/38 file after the snapshot 2024-03-27T06:17:07.848 INFO:tasks.workunit.client.0.smithi132.stdout: create 154/39 file after the snapshot 2024-03-27T06:17:07.849 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/.snap/snap-subdir-test 2024-03-27T06:17:07.889 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/30 file after the snapshot 2024-03-27T06:17:07.891 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/31 file after the snapshot 2024-03-27T06:17:07.893 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/32 file after the snapshot 2024-03-27T06:17:07.895 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/33 file after the snapshot 2024-03-27T06:17:07.897 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/34 file after the snapshot 2024-03-27T06:17:07.898 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/35 file after the snapshot 2024-03-27T06:17:07.900 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/36 file after the snapshot 2024-03-27T06:17:07.902 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/37 file after the snapshot 2024-03-27T06:17:07.904 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/38 file after the snapshot 2024-03-27T06:17:07.905 INFO:tasks.workunit.client.0.smithi132.stdout: create 155/39 file after the snapshot 2024-03-27T06:17:07.907 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/.snap/snap-subdir-test 2024-03-27T06:17:07.948 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/30 file after the snapshot 2024-03-27T06:17:07.950 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/31 file after the snapshot 2024-03-27T06:17:07.952 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/32 file after the snapshot 2024-03-27T06:17:07.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/33 file after the snapshot 2024-03-27T06:17:07.954 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/34 file after the snapshot 2024-03-27T06:17:07.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/35 file after the snapshot 2024-03-27T06:17:07.957 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/36 file after the snapshot 2024-03-27T06:17:07.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/37 file after the snapshot 2024-03-27T06:17:07.959 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/38 file after the snapshot 2024-03-27T06:17:07.961 INFO:tasks.workunit.client.0.smithi132.stdout: create 156/39 file after the snapshot 2024-03-27T06:17:07.962 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/.snap/snap-subdir-test 2024-03-27T06:17:07.995 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/30 file after the snapshot 2024-03-27T06:17:07.996 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/31 file after the snapshot 2024-03-27T06:17:07.997 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/32 file after the snapshot 2024-03-27T06:17:07.998 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/33 file after the snapshot 2024-03-27T06:17:07.999 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/34 file after the snapshot 2024-03-27T06:17:08.000 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/35 file after the snapshot 2024-03-27T06:17:08.002 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/36 file after the snapshot 2024-03-27T06:17:08.003 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/37 file after the snapshot 2024-03-27T06:17:08.004 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/38 file after the snapshot 2024-03-27T06:17:08.006 INFO:tasks.workunit.client.0.smithi132.stdout: create 157/39 file after the snapshot 2024-03-27T06:17:08.007 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/.snap/snap-subdir-test 2024-03-27T06:17:08.041 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/30 file after the snapshot 2024-03-27T06:17:08.043 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/31 file after the snapshot 2024-03-27T06:17:08.044 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/32 file after the snapshot 2024-03-27T06:17:08.046 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/33 file after the snapshot 2024-03-27T06:17:08.047 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/34 file after the snapshot 2024-03-27T06:17:08.049 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/35 file after the snapshot 2024-03-27T06:17:08.050 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/36 file after the snapshot 2024-03-27T06:17:08.052 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/37 file after the snapshot 2024-03-27T06:17:08.054 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/38 file after the snapshot 2024-03-27T06:17:08.055 INFO:tasks.workunit.client.0.smithi132.stdout: create 158/39 file after the snapshot 2024-03-27T06:17:08.057 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/.snap/snap-subdir-test 2024-03-27T06:17:08.092 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/30 file after the snapshot 2024-03-27T06:17:08.094 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/31 file after the snapshot 2024-03-27T06:17:08.095 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/32 file after the snapshot 2024-03-27T06:17:08.097 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/33 file after the snapshot 2024-03-27T06:17:08.099 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/34 file after the snapshot 2024-03-27T06:17:08.100 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/35 file after the snapshot 2024-03-27T06:17:08.102 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/36 file after the snapshot 2024-03-27T06:17:08.104 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/37 file after the snapshot 2024-03-27T06:17:08.105 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/38 file after the snapshot 2024-03-27T06:17:08.107 INFO:tasks.workunit.client.0.smithi132.stdout: create 159/39 file after the snapshot 2024-03-27T06:17:08.109 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/.snap/snap-subdir-test 2024-03-27T06:17:08.143 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/30 file after the snapshot 2024-03-27T06:17:08.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/31 file after the snapshot 2024-03-27T06:17:08.145 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/32 file after the snapshot 2024-03-27T06:17:08.147 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/33 file after the snapshot 2024-03-27T06:17:08.148 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/34 file after the snapshot 2024-03-27T06:17:08.149 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/35 file after the snapshot 2024-03-27T06:17:08.150 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/36 file after the snapshot 2024-03-27T06:17:08.151 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/37 file after the snapshot 2024-03-27T06:17:08.152 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/38 file after the snapshot 2024-03-27T06:17:08.154 INFO:tasks.workunit.client.0.smithi132.stdout: create 160/39 file after the snapshot 2024-03-27T06:17:08.155 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/.snap/snap-subdir-test 2024-03-27T06:17:08.194 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/30 file after the snapshot 2024-03-27T06:17:08.196 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/31 file after the snapshot 2024-03-27T06:17:08.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/32 file after the snapshot 2024-03-27T06:17:08.198 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/33 file after the snapshot 2024-03-27T06:17:08.200 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/34 file after the snapshot 2024-03-27T06:17:08.201 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/35 file after the snapshot 2024-03-27T06:17:08.203 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/36 file after the snapshot 2024-03-27T06:17:08.204 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/37 file after the snapshot 2024-03-27T06:17:08.205 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/38 file after the snapshot 2024-03-27T06:17:08.206 INFO:tasks.workunit.client.0.smithi132.stdout: create 161/39 file after the snapshot 2024-03-27T06:17:08.207 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/.snap/snap-subdir-test 2024-03-27T06:17:08.247 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/30 file after the snapshot 2024-03-27T06:17:08.249 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/31 file after the snapshot 2024-03-27T06:17:08.251 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/32 file after the snapshot 2024-03-27T06:17:08.253 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/33 file after the snapshot 2024-03-27T06:17:08.254 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/34 file after the snapshot 2024-03-27T06:17:08.256 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/35 file after the snapshot 2024-03-27T06:17:08.258 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/36 file after the snapshot 2024-03-27T06:17:08.259 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/37 file after the snapshot 2024-03-27T06:17:08.261 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/38 file after the snapshot 2024-03-27T06:17:08.263 INFO:tasks.workunit.client.0.smithi132.stdout: create 162/39 file after the snapshot 2024-03-27T06:17:08.265 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/.snap/snap-subdir-test 2024-03-27T06:17:08.299 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/30 file after the snapshot 2024-03-27T06:17:08.301 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/31 file after the snapshot 2024-03-27T06:17:08.303 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/32 file after the snapshot 2024-03-27T06:17:08.305 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/33 file after the snapshot 2024-03-27T06:17:08.307 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/34 file after the snapshot 2024-03-27T06:17:08.310 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/35 file after the snapshot 2024-03-27T06:17:08.311 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/36 file after the snapshot 2024-03-27T06:17:08.313 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/37 file after the snapshot 2024-03-27T06:17:08.314 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/38 file after the snapshot 2024-03-27T06:17:08.316 INFO:tasks.workunit.client.0.smithi132.stdout: create 163/39 file after the snapshot 2024-03-27T06:17:08.318 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/.snap/snap-subdir-test 2024-03-27T06:17:08.352 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/30 file after the snapshot 2024-03-27T06:17:08.354 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/31 file after the snapshot 2024-03-27T06:17:08.356 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/32 file after the snapshot 2024-03-27T06:17:08.358 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/33 file after the snapshot 2024-03-27T06:17:08.360 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/34 file after the snapshot 2024-03-27T06:17:08.362 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/35 file after the snapshot 2024-03-27T06:17:08.364 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/36 file after the snapshot 2024-03-27T06:17:08.365 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/37 file after the snapshot 2024-03-27T06:17:08.367 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/38 file after the snapshot 2024-03-27T06:17:08.369 INFO:tasks.workunit.client.0.smithi132.stdout: create 164/39 file after the snapshot 2024-03-27T06:17:08.371 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/.snap/snap-subdir-test 2024-03-27T06:17:08.414 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/30 file after the snapshot 2024-03-27T06:17:08.415 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/31 file after the snapshot 2024-03-27T06:17:08.417 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/32 file after the snapshot 2024-03-27T06:17:08.418 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/33 file after the snapshot 2024-03-27T06:17:08.419 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/34 file after the snapshot 2024-03-27T06:17:08.420 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/35 file after the snapshot 2024-03-27T06:17:08.421 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/36 file after the snapshot 2024-03-27T06:17:08.422 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/37 file after the snapshot 2024-03-27T06:17:08.424 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/38 file after the snapshot 2024-03-27T06:17:08.425 INFO:tasks.workunit.client.0.smithi132.stdout: create 165/39 file after the snapshot 2024-03-27T06:17:08.426 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/.snap/snap-subdir-test 2024-03-27T06:17:08.460 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/30 file after the snapshot 2024-03-27T06:17:08.462 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/31 file after the snapshot 2024-03-27T06:17:08.463 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/32 file after the snapshot 2024-03-27T06:17:08.465 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/33 file after the snapshot 2024-03-27T06:17:08.467 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/34 file after the snapshot 2024-03-27T06:17:08.468 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/35 file after the snapshot 2024-03-27T06:17:08.470 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/36 file after the snapshot 2024-03-27T06:17:08.472 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/37 file after the snapshot 2024-03-27T06:17:08.474 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/38 file after the snapshot 2024-03-27T06:17:08.475 INFO:tasks.workunit.client.0.smithi132.stdout: create 166/39 file after the snapshot 2024-03-27T06:17:08.477 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/.snap/snap-subdir-test 2024-03-27T06:17:08.513 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/30 file after the snapshot 2024-03-27T06:17:08.514 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/31 file after the snapshot 2024-03-27T06:17:08.516 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/32 file after the snapshot 2024-03-27T06:17:08.517 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/33 file after the snapshot 2024-03-27T06:17:08.518 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/34 file after the snapshot 2024-03-27T06:17:08.520 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/35 file after the snapshot 2024-03-27T06:17:08.522 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/36 file after the snapshot 2024-03-27T06:17:08.524 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/37 file after the snapshot 2024-03-27T06:17:08.526 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/38 file after the snapshot 2024-03-27T06:17:08.528 INFO:tasks.workunit.client.0.smithi132.stdout: create 167/39 file after the snapshot 2024-03-27T06:17:08.530 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/.snap/snap-subdir-test 2024-03-27T06:17:08.563 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/30 file after the snapshot 2024-03-27T06:17:08.565 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/31 file after the snapshot 2024-03-27T06:17:08.566 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/32 file after the snapshot 2024-03-27T06:17:08.568 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/33 file after the snapshot 2024-03-27T06:17:08.569 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/34 file after the snapshot 2024-03-27T06:17:08.570 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/35 file after the snapshot 2024-03-27T06:17:08.572 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/36 file after the snapshot 2024-03-27T06:17:08.574 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/37 file after the snapshot 2024-03-27T06:17:08.576 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/38 file after the snapshot 2024-03-27T06:17:08.577 INFO:tasks.workunit.client.0.smithi132.stdout: create 168/39 file after the snapshot 2024-03-27T06:17:08.579 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/.snap/snap-subdir-test 2024-03-27T06:17:08.613 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/30 file after the snapshot 2024-03-27T06:17:08.615 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/31 file after the snapshot 2024-03-27T06:17:08.616 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/32 file after the snapshot 2024-03-27T06:17:08.618 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/33 file after the snapshot 2024-03-27T06:17:08.619 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/34 file after the snapshot 2024-03-27T06:17:08.621 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/35 file after the snapshot 2024-03-27T06:17:08.622 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/36 file after the snapshot 2024-03-27T06:17:08.623 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/37 file after the snapshot 2024-03-27T06:17:08.625 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/38 file after the snapshot 2024-03-27T06:17:08.626 INFO:tasks.workunit.client.0.smithi132.stdout: create 169/39 file after the snapshot 2024-03-27T06:17:08.627 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/.snap/snap-subdir-test 2024-03-27T06:17:08.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/30 file after the snapshot 2024-03-27T06:17:08.665 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/31 file after the snapshot 2024-03-27T06:17:08.667 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/32 file after the snapshot 2024-03-27T06:17:08.668 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/33 file after the snapshot 2024-03-27T06:17:08.670 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/34 file after the snapshot 2024-03-27T06:17:08.672 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/35 file after the snapshot 2024-03-27T06:17:08.673 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/36 file after the snapshot 2024-03-27T06:17:08.675 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/37 file after the snapshot 2024-03-27T06:17:08.677 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/38 file after the snapshot 2024-03-27T06:17:08.678 INFO:tasks.workunit.client.0.smithi132.stdout: create 170/39 file after the snapshot 2024-03-27T06:17:08.680 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/.snap/snap-subdir-test 2024-03-27T06:17:08.715 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/30 file after the snapshot 2024-03-27T06:17:08.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/31 file after the snapshot 2024-03-27T06:17:08.718 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/32 file after the snapshot 2024-03-27T06:17:08.719 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/33 file after the snapshot 2024-03-27T06:17:08.720 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/34 file after the snapshot 2024-03-27T06:17:08.721 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/35 file after the snapshot 2024-03-27T06:17:08.723 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/36 file after the snapshot 2024-03-27T06:17:08.724 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/37 file after the snapshot 2024-03-27T06:17:08.725 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/38 file after the snapshot 2024-03-27T06:17:08.727 INFO:tasks.workunit.client.0.smithi132.stdout: create 171/39 file after the snapshot 2024-03-27T06:17:08.728 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/.snap/snap-subdir-test 2024-03-27T06:17:08.764 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/30 file after the snapshot 2024-03-27T06:17:08.765 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/31 file after the snapshot 2024-03-27T06:17:08.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/32 file after the snapshot 2024-03-27T06:17:08.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/33 file after the snapshot 2024-03-27T06:17:08.769 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/34 file after the snapshot 2024-03-27T06:17:08.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/35 file after the snapshot 2024-03-27T06:17:08.772 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/36 file after the snapshot 2024-03-27T06:17:08.773 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/37 file after the snapshot 2024-03-27T06:17:08.774 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/38 file after the snapshot 2024-03-27T06:17:08.776 INFO:tasks.workunit.client.0.smithi132.stdout: create 172/39 file after the snapshot 2024-03-27T06:17:08.777 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/.snap/snap-subdir-test 2024-03-27T06:17:08.816 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/30 file after the snapshot 2024-03-27T06:17:08.818 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/31 file after the snapshot 2024-03-27T06:17:08.819 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/32 file after the snapshot 2024-03-27T06:17:08.821 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/33 file after the snapshot 2024-03-27T06:17:08.823 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/34 file after the snapshot 2024-03-27T06:17:08.825 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/35 file after the snapshot 2024-03-27T06:17:08.827 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/36 file after the snapshot 2024-03-27T06:17:08.829 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/37 file after the snapshot 2024-03-27T06:17:08.831 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/38 file after the snapshot 2024-03-27T06:17:08.832 INFO:tasks.workunit.client.0.smithi132.stdout: create 173/39 file after the snapshot 2024-03-27T06:17:08.834 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/.snap/snap-subdir-test 2024-03-27T06:17:08.874 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/30 file after the snapshot 2024-03-27T06:17:08.876 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/31 file after the snapshot 2024-03-27T06:17:08.878 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/32 file after the snapshot 2024-03-27T06:17:08.880 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/33 file after the snapshot 2024-03-27T06:17:08.882 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/34 file after the snapshot 2024-03-27T06:17:08.884 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/35 file after the snapshot 2024-03-27T06:17:08.886 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/36 file after the snapshot 2024-03-27T06:17:08.888 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/37 file after the snapshot 2024-03-27T06:17:08.889 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/38 file after the snapshot 2024-03-27T06:17:08.891 INFO:tasks.workunit.client.0.smithi132.stdout: create 174/39 file after the snapshot 2024-03-27T06:17:08.893 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/.snap/snap-subdir-test 2024-03-27T06:17:08.935 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/30 file after the snapshot 2024-03-27T06:17:08.936 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/31 file after the snapshot 2024-03-27T06:17:08.937 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/32 file after the snapshot 2024-03-27T06:17:08.939 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/33 file after the snapshot 2024-03-27T06:17:08.940 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/34 file after the snapshot 2024-03-27T06:17:08.942 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/35 file after the snapshot 2024-03-27T06:17:08.943 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/36 file after the snapshot 2024-03-27T06:17:08.944 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/37 file after the snapshot 2024-03-27T06:17:08.946 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/38 file after the snapshot 2024-03-27T06:17:08.947 INFO:tasks.workunit.client.0.smithi132.stdout: create 175/39 file after the snapshot 2024-03-27T06:17:08.949 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/.snap/snap-subdir-test 2024-03-27T06:17:08.990 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/30 file after the snapshot 2024-03-27T06:17:08.992 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/31 file after the snapshot 2024-03-27T06:17:08.993 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/32 file after the snapshot 2024-03-27T06:17:08.995 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/33 file after the snapshot 2024-03-27T06:17:08.997 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/34 file after the snapshot 2024-03-27T06:17:08.999 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/35 file after the snapshot 2024-03-27T06:17:09.001 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/36 file after the snapshot 2024-03-27T06:17:09.003 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/37 file after the snapshot 2024-03-27T06:17:09.005 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/38 file after the snapshot 2024-03-27T06:17:09.006 INFO:tasks.workunit.client.0.smithi132.stdout: create 176/39 file after the snapshot 2024-03-27T06:17:09.008 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/.snap/snap-subdir-test 2024-03-27T06:17:09.045 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/30 file after the snapshot 2024-03-27T06:17:09.047 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/31 file after the snapshot 2024-03-27T06:17:09.049 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/32 file after the snapshot 2024-03-27T06:17:09.051 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/33 file after the snapshot 2024-03-27T06:17:09.053 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/34 file after the snapshot 2024-03-27T06:17:09.055 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/35 file after the snapshot 2024-03-27T06:17:09.056 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/36 file after the snapshot 2024-03-27T06:17:09.058 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/37 file after the snapshot 2024-03-27T06:17:09.060 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/38 file after the snapshot 2024-03-27T06:17:09.062 INFO:tasks.workunit.client.0.smithi132.stdout: create 177/39 file after the snapshot 2024-03-27T06:17:09.063 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/.snap/snap-subdir-test 2024-03-27T06:17:09.108 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/30 file after the snapshot 2024-03-27T06:17:09.108 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/31 file after the snapshot 2024-03-27T06:17:09.109 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/32 file after the snapshot 2024-03-27T06:17:09.110 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/33 file after the snapshot 2024-03-27T06:17:09.112 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/34 file after the snapshot 2024-03-27T06:17:09.113 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/35 file after the snapshot 2024-03-27T06:17:09.114 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/36 file after the snapshot 2024-03-27T06:17:09.116 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/37 file after the snapshot 2024-03-27T06:17:09.117 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/38 file after the snapshot 2024-03-27T06:17:09.121 INFO:tasks.workunit.client.0.smithi132.stdout: create 178/39 file after the snapshot 2024-03-27T06:17:09.121 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/.snap/snap-subdir-test 2024-03-27T06:17:09.159 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/30 file after the snapshot 2024-03-27T06:17:09.160 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/31 file after the snapshot 2024-03-27T06:17:09.161 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/32 file after the snapshot 2024-03-27T06:17:09.163 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/33 file after the snapshot 2024-03-27T06:17:09.164 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/34 file after the snapshot 2024-03-27T06:17:09.165 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/35 file after the snapshot 2024-03-27T06:17:09.166 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/36 file after the snapshot 2024-03-27T06:17:09.168 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/37 file after the snapshot 2024-03-27T06:17:09.169 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/38 file after the snapshot 2024-03-27T06:17:09.170 INFO:tasks.workunit.client.0.smithi132.stdout: create 179/39 file after the snapshot 2024-03-27T06:17:09.172 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/.snap/snap-subdir-test 2024-03-27T06:17:09.212 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/30 file after the snapshot 2024-03-27T06:17:09.213 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/31 file after the snapshot 2024-03-27T06:17:09.214 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/32 file after the snapshot 2024-03-27T06:17:09.216 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/33 file after the snapshot 2024-03-27T06:17:09.217 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/34 file after the snapshot 2024-03-27T06:17:09.218 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/35 file after the snapshot 2024-03-27T06:17:09.219 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/36 file after the snapshot 2024-03-27T06:17:09.221 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/37 file after the snapshot 2024-03-27T06:17:09.222 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/38 file after the snapshot 2024-03-27T06:17:09.223 INFO:tasks.workunit.client.0.smithi132.stdout: create 180/39 file after the snapshot 2024-03-27T06:17:09.225 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/.snap/snap-subdir-test 2024-03-27T06:17:09.268 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/30 file after the snapshot 2024-03-27T06:17:09.269 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/31 file after the snapshot 2024-03-27T06:17:09.272 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/32 file after the snapshot 2024-03-27T06:17:09.273 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/33 file after the snapshot 2024-03-27T06:17:09.275 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/34 file after the snapshot 2024-03-27T06:17:09.277 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/35 file after the snapshot 2024-03-27T06:17:09.279 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/36 file after the snapshot 2024-03-27T06:17:09.280 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/37 file after the snapshot 2024-03-27T06:17:09.282 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/38 file after the snapshot 2024-03-27T06:17:09.284 INFO:tasks.workunit.client.0.smithi132.stdout: create 181/39 file after the snapshot 2024-03-27T06:17:09.286 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/.snap/snap-subdir-test 2024-03-27T06:17:09.325 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/30 file after the snapshot 2024-03-27T06:17:09.326 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/31 file after the snapshot 2024-03-27T06:17:09.328 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/32 file after the snapshot 2024-03-27T06:17:09.329 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/33 file after the snapshot 2024-03-27T06:17:09.330 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/34 file after the snapshot 2024-03-27T06:17:09.332 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/35 file after the snapshot 2024-03-27T06:17:09.333 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/36 file after the snapshot 2024-03-27T06:17:09.335 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/37 file after the snapshot 2024-03-27T06:17:09.336 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/38 file after the snapshot 2024-03-27T06:17:09.337 INFO:tasks.workunit.client.0.smithi132.stdout: create 182/39 file after the snapshot 2024-03-27T06:17:09.339 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/.snap/snap-subdir-test 2024-03-27T06:17:09.375 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/30 file after the snapshot 2024-03-27T06:17:09.376 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/31 file after the snapshot 2024-03-27T06:17:09.378 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/32 file after the snapshot 2024-03-27T06:17:09.380 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/33 file after the snapshot 2024-03-27T06:17:09.381 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/34 file after the snapshot 2024-03-27T06:17:09.383 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/35 file after the snapshot 2024-03-27T06:17:09.384 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/36 file after the snapshot 2024-03-27T06:17:09.385 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/37 file after the snapshot 2024-03-27T06:17:09.387 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/38 file after the snapshot 2024-03-27T06:17:09.388 INFO:tasks.workunit.client.0.smithi132.stdout: create 183/39 file after the snapshot 2024-03-27T06:17:09.389 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/.snap/snap-subdir-test 2024-03-27T06:17:09.433 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/30 file after the snapshot 2024-03-27T06:17:09.435 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/31 file after the snapshot 2024-03-27T06:17:09.436 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/32 file after the snapshot 2024-03-27T06:17:09.437 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/33 file after the snapshot 2024-03-27T06:17:09.439 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/34 file after the snapshot 2024-03-27T06:17:09.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/35 file after the snapshot 2024-03-27T06:17:09.441 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/36 file after the snapshot 2024-03-27T06:17:09.443 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/37 file after the snapshot 2024-03-27T06:17:09.444 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/38 file after the snapshot 2024-03-27T06:17:09.445 INFO:tasks.workunit.client.0.smithi132.stdout: create 184/39 file after the snapshot 2024-03-27T06:17:09.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/.snap/snap-subdir-test 2024-03-27T06:17:09.488 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/30 file after the snapshot 2024-03-27T06:17:09.489 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/31 file after the snapshot 2024-03-27T06:17:09.490 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/32 file after the snapshot 2024-03-27T06:17:09.492 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/33 file after the snapshot 2024-03-27T06:17:09.493 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/34 file after the snapshot 2024-03-27T06:17:09.494 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/35 file after the snapshot 2024-03-27T06:17:09.496 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/36 file after the snapshot 2024-03-27T06:17:09.498 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/37 file after the snapshot 2024-03-27T06:17:09.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/38 file after the snapshot 2024-03-27T06:17:09.501 INFO:tasks.workunit.client.0.smithi132.stdout: create 185/39 file after the snapshot 2024-03-27T06:17:09.503 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/.snap/snap-subdir-test 2024-03-27T06:17:09.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:09.544 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/30 file after the snapshot 2024-03-27T06:17:09.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/31 file after the snapshot 2024-03-27T06:17:09.546 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/32 file after the snapshot 2024-03-27T06:17:09.547 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/33 file after the snapshot 2024-03-27T06:17:09.548 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/34 file after the snapshot 2024-03-27T06:17:09.549 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/35 file after the snapshot 2024-03-27T06:17:09.550 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/36 file after the snapshot 2024-03-27T06:17:09.551 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/37 file after the snapshot 2024-03-27T06:17:09.553 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/38 file after the snapshot 2024-03-27T06:17:09.554 INFO:tasks.workunit.client.0.smithi132.stdout: create 186/39 file after the snapshot 2024-03-27T06:17:09.556 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/.snap/snap-subdir-test 2024-03-27T06:17:09.592 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/30 file after the snapshot 2024-03-27T06:17:09.593 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/31 file after the snapshot 2024-03-27T06:17:09.594 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/32 file after the snapshot 2024-03-27T06:17:09.595 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/33 file after the snapshot 2024-03-27T06:17:09.596 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/34 file after the snapshot 2024-03-27T06:17:09.597 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/35 file after the snapshot 2024-03-27T06:17:09.597 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/36 file after the snapshot 2024-03-27T06:17:09.599 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/37 file after the snapshot 2024-03-27T06:17:09.600 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/38 file after the snapshot 2024-03-27T06:17:09.601 INFO:tasks.workunit.client.0.smithi132.stdout: create 187/39 file after the snapshot 2024-03-27T06:17:09.603 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/.snap/snap-subdir-test 2024-03-27T06:17:09.640 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/30 file after the snapshot 2024-03-27T06:17:09.642 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/31 file after the snapshot 2024-03-27T06:17:09.643 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/32 file after the snapshot 2024-03-27T06:17:09.644 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/33 file after the snapshot 2024-03-27T06:17:09.646 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/34 file after the snapshot 2024-03-27T06:17:09.647 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/35 file after the snapshot 2024-03-27T06:17:09.648 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/36 file after the snapshot 2024-03-27T06:17:09.650 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/37 file after the snapshot 2024-03-27T06:17:09.651 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/38 file after the snapshot 2024-03-27T06:17:09.652 INFO:tasks.workunit.client.0.smithi132.stdout: create 188/39 file after the snapshot 2024-03-27T06:17:09.654 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/.snap/snap-subdir-test 2024-03-27T06:17:09.693 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/30 file after the snapshot 2024-03-27T06:17:09.695 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/31 file after the snapshot 2024-03-27T06:17:09.697 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/32 file after the snapshot 2024-03-27T06:17:09.698 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/33 file after the snapshot 2024-03-27T06:17:09.700 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/34 file after the snapshot 2024-03-27T06:17:09.701 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/35 file after the snapshot 2024-03-27T06:17:09.703 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/36 file after the snapshot 2024-03-27T06:17:09.704 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/37 file after the snapshot 2024-03-27T06:17:09.706 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/38 file after the snapshot 2024-03-27T06:17:09.707 INFO:tasks.workunit.client.0.smithi132.stdout: create 189/39 file after the snapshot 2024-03-27T06:17:09.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.717+0000 7fb618400640 1 Processor -- start 2024-03-27T06:17:09.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.717+0000 7fb618400640 1 -- start start 2024-03-27T06:17:09.709 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/.snap/snap-subdir-test 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.718+0000 7fb618400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb610150bd0 0x7fb6101569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.718+0000 7fb618400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb6100c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.718+0000 7fb618400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb61015aa70 con 0x7fb6100713e0 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.718+0000 7fb618400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fb6100c4310 con 0x7fb610150bd0 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.718+0000 7fb618400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb6100c8640 con 0x7fb6100c9370 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb616400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb6100c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb615000640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fb6100713e0 0x7fb610159fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:46490/0 (socket says 172.21.15.132:46490) 2024-03-27T06:17:09.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb615000640 1 -- 172.21.15.132:0/3845177904 learned_addr learned my addr 172.21.15.132:0/3845177904 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:17:09.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb615a00640 1 --2- 172.21.15.132:0/3845177904 >> v2:172.21.15.132:3300/0 conn(0x7fb610150bd0 0x7fb6101569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb616400640 1 -- 172.21.15.132:0/3845177904 >> v2:172.21.15.132:3300/0 conn(0x7fb610150bd0 msgr2=0x7fb6101569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb616400640 1 --2- 172.21.15.132:0/3845177904 >> v2:172.21.15.132:3300/0 conn(0x7fb610150bd0 0x7fb6101569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb616400640 1 -- 172.21.15.132:0/3845177904 >> v1:172.21.15.132:6789/0 conn(0x7fb6100713e0 legacy=0x7fb610159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb616400640 1 -- 172.21.15.132:0/3845177904 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61015ad30 con 0x7fb6100c9370 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.719+0000 7fb60fe00640 1 -- 172.21.15.132:0/3845177904 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889649444 0 0) 0x7fb61015aa70 con 0x7fb6100713e0 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.720+0000 7fb616400640 1 --2- 172.21.15.132:0/3845177904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb6100c73b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb608009d90 tx=0x7fb60802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195081da03b872a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:09.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb60fe00640 1 -- 172.21.15.132:0/3845177904 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608039070 con 0x7fb6100c9370 2024-03-27T06:17:09.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb60fe00640 1 -- 172.21.15.132:0/3845177904 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb60802cc60 con 0x7fb6100c9370 2024-03-27T06:17:09.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb60fe00640 1 -- 172.21.15.132:0/3845177904 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608034660 con 0x7fb6100c9370 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb618400640 1 -- 172.21.15.132:0/3845177904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 msgr2=0x7fb6100c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb618400640 1 --2- 172.21.15.132:0/3845177904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb6100c73b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb608009d90 tx=0x7fb60802beb0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb618400640 1 -- 172.21.15.132:0/3845177904 shutdown_connections 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb618400640 1 --2- 172.21.15.132:0/3845177904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb6100c73b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.721+0000 7fb618400640 1 --2- 172.21.15.132:0/3845177904 >> v2:172.21.15.132:3300/0 conn(0x7fb610150bd0 0x7fb6101569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.722+0000 7fb618400640 1 -- 172.21.15.132:0/3845177904 >> 172.21.15.132:0/3845177904 conn(0x7fb61006eef0 msgr2=0x7fb61006f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.722+0000 7fb618400640 1 -- 172.21.15.132:0/3845177904 shutdown_connections 2024-03-27T06:17:09.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.722+0000 7fb618400640 1 -- 172.21.15.132:0/3845177904 wait complete. 2024-03-27T06:17:09.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.724+0000 7fb618400640 1 Processor -- start 2024-03-27T06:17:09.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.725+0000 7fb618400640 1 -- start start 2024-03-27T06:17:09.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb610151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb610150bd0 0x7fb610151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb6100c8b70 con 0x7fb610150bd0 2024-03-27T06:17:09.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb6100c9800 con 0x7fb6100713e0 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb618400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb6100c8e50 con 0x7fb6100c9370 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb615a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb615000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb610151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb615a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:51546/0 (socket says 172.21.15.132:51546) 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb615a00640 1 -- 172.21.15.132:0/1680033538 learned_addr learned my addr 172.21.15.132:0/1680033538 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.726+0000 7fb616400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb610150bd0 0x7fb610151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 -- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 msgr2=0x7fb610151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb610151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 -- 172.21.15.132:0/1680033538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb610150bd0 msgr2=0x7fb610151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb610150bd0 0x7fb610151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 -- 172.21.15.132:0/1680033538 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61014e8f0 con 0x7fb6100713e0 2024-03-27T06:17:09.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615000640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb610151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb615a00640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e870 tx=0x7fb60400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604017070 con 0x7fb6100713e0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb60400c940 con 0x7fb6100713e0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60400cd30 con 0x7fb6100713e0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.727+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb610153780 con 0x7fb6100713e0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.728+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb610153d40 con 0x7fb6100713e0 2024-03-27T06:17:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.729+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb610068840 con 0x7fb6100713e0 2024-03-27T06:17:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.730+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb60401a070 con 0x7fb6100713e0 2024-03-27T06:17:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.731+0000 7fb60d600640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 0x7fb5f006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.731+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5910+0+0 (secure 0 0 0) 0x7fb60408b920 con 0x7fb6100713e0 2024-03-27T06:17:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.731+0000 7fb615000640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 0x7fb5f006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.732+0000 7fb615000640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 0x7fb5f006ea20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb610158aa0 tx=0x7fb5fc007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:09.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.738+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb604059810 con 0x7fb6100713e0 2024-03-27T06:17:09.744 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/30 file after the snapshot 2024-03-27T06:17:09.745 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/31 file after the snapshot 2024-03-27T06:17:09.746 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/32 file after the snapshot 2024-03-27T06:17:09.747 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/33 file after the snapshot 2024-03-27T06:17:09.747 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/34 file after the snapshot 2024-03-27T06:17:09.748 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/35 file after the snapshot 2024-03-27T06:17:09.750 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/36 file after the snapshot 2024-03-27T06:17:09.752 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/37 file after the snapshot 2024-03-27T06:17:09.753 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/38 file after the snapshot 2024-03-27T06:17:09.755 INFO:tasks.workunit.client.0.smithi132.stdout: create 190/39 file after the snapshot 2024-03-27T06:17:09.756 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/.snap/snap-subdir-test 2024-03-27T06:17:09.794 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/30 file after the snapshot 2024-03-27T06:17:09.795 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/31 file after the snapshot 2024-03-27T06:17:09.797 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/32 file after the snapshot 2024-03-27T06:17:09.798 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/33 file after the snapshot 2024-03-27T06:17:09.799 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/34 file after the snapshot 2024-03-27T06:17:09.800 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/35 file after the snapshot 2024-03-27T06:17:09.801 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/36 file after the snapshot 2024-03-27T06:17:09.803 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/37 file after the snapshot 2024-03-27T06:17:09.804 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/38 file after the snapshot 2024-03-27T06:17:09.805 INFO:tasks.workunit.client.0.smithi132.stdout: create 191/39 file after the snapshot 2024-03-27T06:17:09.807 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/.snap/snap-subdir-test 2024-03-27T06:17:09.844 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/30 file after the snapshot 2024-03-27T06:17:09.845 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/31 file after the snapshot 2024-03-27T06:17:09.846 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/32 file after the snapshot 2024-03-27T06:17:09.848 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/33 file after the snapshot 2024-03-27T06:17:09.849 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/34 file after the snapshot 2024-03-27T06:17:09.850 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/35 file after the snapshot 2024-03-27T06:17:09.852 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/36 file after the snapshot 2024-03-27T06:17:09.853 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/37 file after the snapshot 2024-03-27T06:17:09.854 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/38 file after the snapshot 2024-03-27T06:17:09.855 INFO:tasks.workunit.client.0.smithi132.stdout: create 192/39 file after the snapshot 2024-03-27T06:17:09.857 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/.snap/snap-subdir-test 2024-03-27T06:17:09.893 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/30 file after the snapshot 2024-03-27T06:17:09.895 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/31 file after the snapshot 2024-03-27T06:17:09.896 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/32 file after the snapshot 2024-03-27T06:17:09.898 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/33 file after the snapshot 2024-03-27T06:17:09.900 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/34 file after the snapshot 2024-03-27T06:17:09.902 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/35 file after the snapshot 2024-03-27T06:17:09.903 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/36 file after the snapshot 2024-03-27T06:17:09.905 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/37 file after the snapshot 2024-03-27T06:17:09.907 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/38 file after the snapshot 2024-03-27T06:17:09.908 INFO:tasks.workunit.client.0.smithi132.stdout: create 193/39 file after the snapshot 2024-03-27T06:17:09.910 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/.snap/snap-subdir-test 2024-03-27T06:17:09.944 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/30 file after the snapshot 2024-03-27T06:17:09.945 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/31 file after the snapshot 2024-03-27T06:17:09.946 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/32 file after the snapshot 2024-03-27T06:17:09.948 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/33 file after the snapshot 2024-03-27T06:17:09.949 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/34 file after the snapshot 2024-03-27T06:17:09.950 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/35 file after the snapshot 2024-03-27T06:17:09.951 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/36 file after the snapshot 2024-03-27T06:17:09.952 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/37 file after the snapshot 2024-03-27T06:17:09.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/38 file after the snapshot 2024-03-27T06:17:09.955 INFO:tasks.workunit.client.0.smithi132.stdout: create 194/39 file after the snapshot 2024-03-27T06:17:09.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/.snap/snap-subdir-test 2024-03-27T06:17:09.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.988+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6100c7ef0 con 0x7fb6100713e0 2024-03-27T06:17:09.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.989+0000 7fb60d600640 1 -- 172.21.15.132:0/1680033538 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7fb60405db10 con 0x7fb6100713e0 2024-03-27T06:17:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:09.980 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:17:09.981 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 msgr2=0x7fb5f006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 0x7fb5f006ea20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb610158aa0 tx=0x7fb5fc007d50 comp rx=0 tx=0).stop 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 msgr2=0x7fb6101f0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e870 tx=0x7fb60400ed30 comp rx=0 tx=0).stop 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 shutdown_connections 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb610150bd0 0x7fb610151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb6100c9370 0x7fb610151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb5f006c560 0x7fb5f006ea20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.991+0000 7fb618400640 1 --2- 172.21.15.132:0/1680033538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6100713e0 0x7fb6101f0b10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:09.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.992+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 >> 172.21.15.132:0/1680033538 conn(0x7fb61006eef0 msgr2=0x7fb61006f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:09.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.992+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 shutdown_connections 2024-03-27T06:17:09.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:09.992+0000 7fb618400640 1 -- 172.21.15.132:0/1680033538 wait complete. 2024-03-27T06:17:10.001 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-03-27T06:17:10.001 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2024-03-27T06:17:10.010 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/30 file after the snapshot 2024-03-27T06:17:10.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/31 file after the snapshot 2024-03-27T06:17:10.012 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/32 file after the snapshot 2024-03-27T06:17:10.014 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/33 file after the snapshot 2024-03-27T06:17:10.015 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/34 file after the snapshot 2024-03-27T06:17:10.016 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/35 file after the snapshot 2024-03-27T06:17:10.017 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/36 file after the snapshot 2024-03-27T06:17:10.017 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/37 file after the snapshot 2024-03-27T06:17:10.019 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/38 file after the snapshot 2024-03-27T06:17:10.020 INFO:tasks.workunit.client.0.smithi132.stdout: create 195/39 file after the snapshot 2024-03-27T06:17:10.022 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/.snap/snap-subdir-test 2024-03-27T06:17:10.069 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/30 file after the snapshot 2024-03-27T06:17:10.071 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/31 file after the snapshot 2024-03-27T06:17:10.072 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/32 file after the snapshot 2024-03-27T06:17:10.074 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/33 file after the snapshot 2024-03-27T06:17:10.076 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/34 file after the snapshot 2024-03-27T06:17:10.077 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/35 file after the snapshot 2024-03-27T06:17:10.079 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/36 file after the snapshot 2024-03-27T06:17:10.081 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/37 file after the snapshot 2024-03-27T06:17:10.082 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/38 file after the snapshot 2024-03-27T06:17:10.084 INFO:tasks.workunit.client.0.smithi132.stdout: create 196/39 file after the snapshot 2024-03-27T06:17:10.086 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/.snap/snap-subdir-test 2024-03-27T06:17:10.133 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/30 file after the snapshot 2024-03-27T06:17:10.134 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/31 file after the snapshot 2024-03-27T06:17:10.136 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/32 file after the snapshot 2024-03-27T06:17:10.137 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/33 file after the snapshot 2024-03-27T06:17:10.138 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/34 file after the snapshot 2024-03-27T06:17:10.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/35 file after the snapshot 2024-03-27T06:17:10.141 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/36 file after the snapshot 2024-03-27T06:17:10.142 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/37 file after the snapshot 2024-03-27T06:17:10.143 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/38 file after the snapshot 2024-03-27T06:17:10.145 INFO:tasks.workunit.client.0.smithi132.stdout: create 197/39 file after the snapshot 2024-03-27T06:17:10.146 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/.snap/snap-subdir-test 2024-03-27T06:17:10.192 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/30 file after the snapshot 2024-03-27T06:17:10.194 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/31 file after the snapshot 2024-03-27T06:17:10.196 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/32 file after the snapshot 2024-03-27T06:17:10.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/33 file after the snapshot 2024-03-27T06:17:10.199 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/34 file after the snapshot 2024-03-27T06:17:10.201 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/35 file after the snapshot 2024-03-27T06:17:10.202 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/36 file after the snapshot 2024-03-27T06:17:10.204 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/37 file after the snapshot 2024-03-27T06:17:10.206 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/38 file after the snapshot 2024-03-27T06:17:10.208 INFO:tasks.workunit.client.0.smithi132.stdout: create 198/39 file after the snapshot 2024-03-27T06:17:10.209 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/.snap/snap-subdir-test 2024-03-27T06:17:10.264 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/30 file after the snapshot 2024-03-27T06:17:10.265 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/31 file after the snapshot 2024-03-27T06:17:10.267 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/32 file after the snapshot 2024-03-27T06:17:10.268 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/33 file after the snapshot 2024-03-27T06:17:10.269 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/34 file after the snapshot 2024-03-27T06:17:10.271 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/35 file after the snapshot 2024-03-27T06:17:10.272 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/36 file after the snapshot 2024-03-27T06:17:10.273 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/37 file after the snapshot 2024-03-27T06:17:10.275 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/38 file after the snapshot 2024-03-27T06:17:10.276 INFO:tasks.workunit.client.0.smithi132.stdout: create 199/39 file after the snapshot 2024-03-27T06:17:10.277 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/.snap/snap-subdir-test 2024-03-27T06:17:10.328 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/30 file after the snapshot 2024-03-27T06:17:10.330 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/31 file after the snapshot 2024-03-27T06:17:10.332 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/32 file after the snapshot 2024-03-27T06:17:10.333 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/33 file after the snapshot 2024-03-27T06:17:10.335 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/34 file after the snapshot 2024-03-27T06:17:10.337 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/35 file after the snapshot 2024-03-27T06:17:10.339 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/36 file after the snapshot 2024-03-27T06:17:10.340 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/37 file after the snapshot 2024-03-27T06:17:10.342 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/38 file after the snapshot 2024-03-27T06:17:10.344 INFO:tasks.workunit.client.0.smithi132.stdout: create 200/39 file after the snapshot 2024-03-27T06:17:10.346 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/.snap/snap-subdir-test 2024-03-27T06:17:10.398 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/30 file after the snapshot 2024-03-27T06:17:10.400 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/31 file after the snapshot 2024-03-27T06:17:10.401 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/32 file after the snapshot 2024-03-27T06:17:10.403 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/33 file after the snapshot 2024-03-27T06:17:10.404 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/34 file after the snapshot 2024-03-27T06:17:10.406 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/35 file after the snapshot 2024-03-27T06:17:10.407 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/36 file after the snapshot 2024-03-27T06:17:10.409 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/37 file after the snapshot 2024-03-27T06:17:10.410 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/38 file after the snapshot 2024-03-27T06:17:10.412 INFO:tasks.workunit.client.0.smithi132.stdout: create 201/39 file after the snapshot 2024-03-27T06:17:10.414 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/.snap/snap-subdir-test 2024-03-27T06:17:10.470 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/30 file after the snapshot 2024-03-27T06:17:10.472 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/31 file after the snapshot 2024-03-27T06:17:10.473 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/32 file after the snapshot 2024-03-27T06:17:10.474 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/33 file after the snapshot 2024-03-27T06:17:10.475 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/34 file after the snapshot 2024-03-27T06:17:10.477 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/35 file after the snapshot 2024-03-27T06:17:10.478 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/36 file after the snapshot 2024-03-27T06:17:10.479 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/37 file after the snapshot 2024-03-27T06:17:10.481 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/38 file after the snapshot 2024-03-27T06:17:10.482 INFO:tasks.workunit.client.0.smithi132.stdout: create 202/39 file after the snapshot 2024-03-27T06:17:10.484 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/.snap/snap-subdir-test 2024-03-27T06:17:10.538 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/30 file after the snapshot 2024-03-27T06:17:10.539 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/31 file after the snapshot 2024-03-27T06:17:10.540 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/32 file after the snapshot 2024-03-27T06:17:10.542 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/33 file after the snapshot 2024-03-27T06:17:10.543 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/34 file after the snapshot 2024-03-27T06:17:10.544 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/35 file after the snapshot 2024-03-27T06:17:10.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/36 file after the snapshot 2024-03-27T06:17:10.547 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/37 file after the snapshot 2024-03-27T06:17:10.548 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/38 file after the snapshot 2024-03-27T06:17:10.549 INFO:tasks.workunit.client.0.smithi132.stdout: create 203/39 file after the snapshot 2024-03-27T06:17:10.551 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/.snap/snap-subdir-test 2024-03-27T06:17:10.601 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/30 file after the snapshot 2024-03-27T06:17:10.602 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/31 file after the snapshot 2024-03-27T06:17:10.604 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/32 file after the snapshot 2024-03-27T06:17:10.605 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/33 file after the snapshot 2024-03-27T06:17:10.606 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/34 file after the snapshot 2024-03-27T06:17:10.607 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/35 file after the snapshot 2024-03-27T06:17:10.609 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/36 file after the snapshot 2024-03-27T06:17:10.610 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/37 file after the snapshot 2024-03-27T06:17:10.611 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/38 file after the snapshot 2024-03-27T06:17:10.613 INFO:tasks.workunit.client.0.smithi132.stdout: create 204/39 file after the snapshot 2024-03-27T06:17:10.614 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/.snap/snap-subdir-test 2024-03-27T06:17:10.657 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/30 file after the snapshot 2024-03-27T06:17:10.659 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/31 file after the snapshot 2024-03-27T06:17:10.661 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/32 file after the snapshot 2024-03-27T06:17:10.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/33 file after the snapshot 2024-03-27T06:17:10.664 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/34 file after the snapshot 2024-03-27T06:17:10.666 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/35 file after the snapshot 2024-03-27T06:17:10.668 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/36 file after the snapshot 2024-03-27T06:17:10.670 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/37 file after the snapshot 2024-03-27T06:17:10.672 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/38 file after the snapshot 2024-03-27T06:17:10.673 INFO:tasks.workunit.client.0.smithi132.stdout: create 205/39 file after the snapshot 2024-03-27T06:17:10.675 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/.snap/snap-subdir-test 2024-03-27T06:17:10.713 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/30 file after the snapshot 2024-03-27T06:17:10.714 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/31 file after the snapshot 2024-03-27T06:17:10.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/32 file after the snapshot 2024-03-27T06:17:10.718 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/33 file after the snapshot 2024-03-27T06:17:10.720 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/34 file after the snapshot 2024-03-27T06:17:10.721 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/35 file after the snapshot 2024-03-27T06:17:10.723 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/36 file after the snapshot 2024-03-27T06:17:10.724 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/37 file after the snapshot 2024-03-27T06:17:10.726 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/38 file after the snapshot 2024-03-27T06:17:10.727 INFO:tasks.workunit.client.0.smithi132.stdout: create 206/39 file after the snapshot 2024-03-27T06:17:10.728 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/.snap/snap-subdir-test 2024-03-27T06:17:10.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/30 file after the snapshot 2024-03-27T06:17:10.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/31 file after the snapshot 2024-03-27T06:17:10.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/32 file after the snapshot 2024-03-27T06:17:10.771 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/33 file after the snapshot 2024-03-27T06:17:10.772 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/34 file after the snapshot 2024-03-27T06:17:10.774 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/35 file after the snapshot 2024-03-27T06:17:10.775 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/36 file after the snapshot 2024-03-27T06:17:10.776 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/37 file after the snapshot 2024-03-27T06:17:10.778 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/38 file after the snapshot 2024-03-27T06:17:10.779 INFO:tasks.workunit.client.0.smithi132.stdout: create 207/39 file after the snapshot 2024-03-27T06:17:10.780 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/.snap/snap-subdir-test 2024-03-27T06:17:10.819 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/30 file after the snapshot 2024-03-27T06:17:10.821 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/31 file after the snapshot 2024-03-27T06:17:10.823 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/32 file after the snapshot 2024-03-27T06:17:10.824 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/33 file after the snapshot 2024-03-27T06:17:10.825 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/34 file after the snapshot 2024-03-27T06:17:10.826 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/35 file after the snapshot 2024-03-27T06:17:10.827 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/36 file after the snapshot 2024-03-27T06:17:10.828 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/37 file after the snapshot 2024-03-27T06:17:10.830 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/38 file after the snapshot 2024-03-27T06:17:10.831 INFO:tasks.workunit.client.0.smithi132.stdout: create 208/39 file after the snapshot 2024-03-27T06:17:10.832 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/.snap/snap-subdir-test 2024-03-27T06:17:10.867 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/30 file after the snapshot 2024-03-27T06:17:10.868 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/31 file after the snapshot 2024-03-27T06:17:10.870 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/32 file after the snapshot 2024-03-27T06:17:10.870 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/33 file after the snapshot 2024-03-27T06:17:10.872 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/34 file after the snapshot 2024-03-27T06:17:10.873 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/35 file after the snapshot 2024-03-27T06:17:10.874 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/36 file after the snapshot 2024-03-27T06:17:10.875 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/37 file after the snapshot 2024-03-27T06:17:10.876 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/38 file after the snapshot 2024-03-27T06:17:10.877 INFO:tasks.workunit.client.0.smithi132.stdout: create 209/39 file after the snapshot 2024-03-27T06:17:10.878 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/.snap/snap-subdir-test 2024-03-27T06:17:10.913 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/30 file after the snapshot 2024-03-27T06:17:10.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/31 file after the snapshot 2024-03-27T06:17:10.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/32 file after the snapshot 2024-03-27T06:17:10.916 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/33 file after the snapshot 2024-03-27T06:17:10.917 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/34 file after the snapshot 2024-03-27T06:17:10.919 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/35 file after the snapshot 2024-03-27T06:17:10.920 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/36 file after the snapshot 2024-03-27T06:17:10.921 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/37 file after the snapshot 2024-03-27T06:17:10.922 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/38 file after the snapshot 2024-03-27T06:17:10.923 INFO:tasks.workunit.client.0.smithi132.stdout: create 210/39 file after the snapshot 2024-03-27T06:17:10.924 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/.snap/snap-subdir-test 2024-03-27T06:17:10.959 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/30 file after the snapshot 2024-03-27T06:17:10.962 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/31 file after the snapshot 2024-03-27T06:17:10.963 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/32 file after the snapshot 2024-03-27T06:17:10.964 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/33 file after the snapshot 2024-03-27T06:17:10.966 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/34 file after the snapshot 2024-03-27T06:17:10.968 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/35 file after the snapshot 2024-03-27T06:17:10.970 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/36 file after the snapshot 2024-03-27T06:17:10.971 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/37 file after the snapshot 2024-03-27T06:17:10.973 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/38 file after the snapshot 2024-03-27T06:17:10.974 INFO:tasks.workunit.client.0.smithi132.stdout: create 211/39 file after the snapshot 2024-03-27T06:17:10.976 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/.snap/snap-subdir-test 2024-03-27T06:17:11.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/30 file after the snapshot 2024-03-27T06:17:11.012 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/31 file after the snapshot 2024-03-27T06:17:11.013 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/32 file after the snapshot 2024-03-27T06:17:11.015 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/33 file after the snapshot 2024-03-27T06:17:11.017 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/34 file after the snapshot 2024-03-27T06:17:11.018 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/35 file after the snapshot 2024-03-27T06:17:11.020 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/36 file after the snapshot 2024-03-27T06:17:11.021 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/37 file after the snapshot 2024-03-27T06:17:11.023 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/38 file after the snapshot 2024-03-27T06:17:11.025 INFO:tasks.workunit.client.0.smithi132.stdout: create 212/39 file after the snapshot 2024-03-27T06:17:11.027 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/.snap/snap-subdir-test 2024-03-27T06:17:11.062 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/30 file after the snapshot 2024-03-27T06:17:11.063 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/31 file after the snapshot 2024-03-27T06:17:11.064 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/32 file after the snapshot 2024-03-27T06:17:11.065 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/33 file after the snapshot 2024-03-27T06:17:11.066 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/34 file after the snapshot 2024-03-27T06:17:11.068 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/35 file after the snapshot 2024-03-27T06:17:11.068 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/36 file after the snapshot 2024-03-27T06:17:11.070 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/37 file after the snapshot 2024-03-27T06:17:11.071 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/38 file after the snapshot 2024-03-27T06:17:11.072 INFO:tasks.workunit.client.0.smithi132.stdout: create 213/39 file after the snapshot 2024-03-27T06:17:11.073 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/.snap/snap-subdir-test 2024-03-27T06:17:11.115 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/30 file after the snapshot 2024-03-27T06:17:11.116 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/31 file after the snapshot 2024-03-27T06:17:11.118 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/32 file after the snapshot 2024-03-27T06:17:11.119 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/33 file after the snapshot 2024-03-27T06:17:11.121 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/34 file after the snapshot 2024-03-27T06:17:11.122 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/35 file after the snapshot 2024-03-27T06:17:11.124 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/36 file after the snapshot 2024-03-27T06:17:11.125 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/37 file after the snapshot 2024-03-27T06:17:11.126 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/38 file after the snapshot 2024-03-27T06:17:11.128 INFO:tasks.workunit.client.0.smithi132.stdout: create 214/39 file after the snapshot 2024-03-27T06:17:11.129 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/.snap/snap-subdir-test 2024-03-27T06:17:11.166 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/30 file after the snapshot 2024-03-27T06:17:11.167 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/31 file after the snapshot 2024-03-27T06:17:11.169 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/32 file after the snapshot 2024-03-27T06:17:11.170 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/33 file after the snapshot 2024-03-27T06:17:11.172 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/34 file after the snapshot 2024-03-27T06:17:11.174 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/35 file after the snapshot 2024-03-27T06:17:11.175 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/36 file after the snapshot 2024-03-27T06:17:11.177 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/37 file after the snapshot 2024-03-27T06:17:11.179 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/38 file after the snapshot 2024-03-27T06:17:11.180 INFO:tasks.workunit.client.0.smithi132.stdout: create 215/39 file after the snapshot 2024-03-27T06:17:11.182 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/.snap/snap-subdir-test 2024-03-27T06:17:11.222 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/30 file after the snapshot 2024-03-27T06:17:11.224 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/31 file after the snapshot 2024-03-27T06:17:11.225 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/32 file after the snapshot 2024-03-27T06:17:11.227 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/33 file after the snapshot 2024-03-27T06:17:11.228 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/34 file after the snapshot 2024-03-27T06:17:11.230 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/35 file after the snapshot 2024-03-27T06:17:11.231 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/36 file after the snapshot 2024-03-27T06:17:11.233 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/37 file after the snapshot 2024-03-27T06:17:11.234 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/38 file after the snapshot 2024-03-27T06:17:11.236 INFO:tasks.workunit.client.0.smithi132.stdout: create 216/39 file after the snapshot 2024-03-27T06:17:11.238 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/.snap/snap-subdir-test 2024-03-27T06:17:11.279 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/30 file after the snapshot 2024-03-27T06:17:11.280 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/31 file after the snapshot 2024-03-27T06:17:11.281 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/32 file after the snapshot 2024-03-27T06:17:11.282 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/33 file after the snapshot 2024-03-27T06:17:11.283 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/34 file after the snapshot 2024-03-27T06:17:11.285 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/35 file after the snapshot 2024-03-27T06:17:11.286 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/36 file after the snapshot 2024-03-27T06:17:11.288 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/37 file after the snapshot 2024-03-27T06:17:11.289 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/38 file after the snapshot 2024-03-27T06:17:11.290 INFO:tasks.workunit.client.0.smithi132.stdout: create 217/39 file after the snapshot 2024-03-27T06:17:11.291 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/.snap/snap-subdir-test 2024-03-27T06:17:11.333 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/30 file after the snapshot 2024-03-27T06:17:11.334 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/31 file after the snapshot 2024-03-27T06:17:11.336 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/32 file after the snapshot 2024-03-27T06:17:11.337 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/33 file after the snapshot 2024-03-27T06:17:11.338 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/34 file after the snapshot 2024-03-27T06:17:11.339 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/35 file after the snapshot 2024-03-27T06:17:11.341 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/36 file after the snapshot 2024-03-27T06:17:11.342 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/37 file after the snapshot 2024-03-27T06:17:11.343 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/38 file after the snapshot 2024-03-27T06:17:11.345 INFO:tasks.workunit.client.0.smithi132.stdout: create 218/39 file after the snapshot 2024-03-27T06:17:11.346 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/.snap/snap-subdir-test 2024-03-27T06:17:11.387 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/30 file after the snapshot 2024-03-27T06:17:11.389 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/31 file after the snapshot 2024-03-27T06:17:11.390 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/32 file after the snapshot 2024-03-27T06:17:11.391 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/33 file after the snapshot 2024-03-27T06:17:11.392 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/34 file after the snapshot 2024-03-27T06:17:11.394 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/35 file after the snapshot 2024-03-27T06:17:11.395 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/36 file after the snapshot 2024-03-27T06:17:11.396 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/37 file after the snapshot 2024-03-27T06:17:11.397 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/38 file after the snapshot 2024-03-27T06:17:11.399 INFO:tasks.workunit.client.0.smithi132.stdout: create 219/39 file after the snapshot 2024-03-27T06:17:11.400 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/.snap/snap-subdir-test 2024-03-27T06:17:11.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/30 file after the snapshot 2024-03-27T06:17:11.442 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/31 file after the snapshot 2024-03-27T06:17:11.443 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/32 file after the snapshot 2024-03-27T06:17:11.444 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/33 file after the snapshot 2024-03-27T06:17:11.445 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/34 file after the snapshot 2024-03-27T06:17:11.446 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/35 file after the snapshot 2024-03-27T06:17:11.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/36 file after the snapshot 2024-03-27T06:17:11.448 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/37 file after the snapshot 2024-03-27T06:17:11.449 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/38 file after the snapshot 2024-03-27T06:17:11.450 INFO:tasks.workunit.client.0.smithi132.stdout: create 220/39 file after the snapshot 2024-03-27T06:17:11.452 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/.snap/snap-subdir-test 2024-03-27T06:17:11.486 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/30 file after the snapshot 2024-03-27T06:17:11.487 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/31 file after the snapshot 2024-03-27T06:17:11.489 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/32 file after the snapshot 2024-03-27T06:17:11.490 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/33 file after the snapshot 2024-03-27T06:17:11.491 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/34 file after the snapshot 2024-03-27T06:17:11.492 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/35 file after the snapshot 2024-03-27T06:17:11.494 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/36 file after the snapshot 2024-03-27T06:17:11.495 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/37 file after the snapshot 2024-03-27T06:17:11.497 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/38 file after the snapshot 2024-03-27T06:17:11.498 INFO:tasks.workunit.client.0.smithi132.stdout: create 221/39 file after the snapshot 2024-03-27T06:17:11.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/.snap/snap-subdir-test 2024-03-27T06:17:11.542 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/30 file after the snapshot 2024-03-27T06:17:11.544 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/31 file after the snapshot 2024-03-27T06:17:11.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/32 file after the snapshot 2024-03-27T06:17:11.547 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/33 file after the snapshot 2024-03-27T06:17:11.549 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/34 file after the snapshot 2024-03-27T06:17:11.551 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/35 file after the snapshot 2024-03-27T06:17:11.552 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/36 file after the snapshot 2024-03-27T06:17:11.554 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/37 file after the snapshot 2024-03-27T06:17:11.555 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/38 file after the snapshot 2024-03-27T06:17:11.557 INFO:tasks.workunit.client.0.smithi132.stdout: create 222/39 file after the snapshot 2024-03-27T06:17:11.558 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/.snap/snap-subdir-test 2024-03-27T06:17:11.597 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/30 file after the snapshot 2024-03-27T06:17:11.598 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/31 file after the snapshot 2024-03-27T06:17:11.599 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/32 file after the snapshot 2024-03-27T06:17:11.600 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/33 file after the snapshot 2024-03-27T06:17:11.601 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/34 file after the snapshot 2024-03-27T06:17:11.603 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/35 file after the snapshot 2024-03-27T06:17:11.604 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/36 file after the snapshot 2024-03-27T06:17:11.605 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/37 file after the snapshot 2024-03-27T06:17:11.607 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/38 file after the snapshot 2024-03-27T06:17:11.608 INFO:tasks.workunit.client.0.smithi132.stdout: create 223/39 file after the snapshot 2024-03-27T06:17:11.609 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/.snap/snap-subdir-test 2024-03-27T06:17:11.649 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/30 file after the snapshot 2024-03-27T06:17:11.651 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/31 file after the snapshot 2024-03-27T06:17:11.653 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/32 file after the snapshot 2024-03-27T06:17:11.654 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/33 file after the snapshot 2024-03-27T06:17:11.656 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/34 file after the snapshot 2024-03-27T06:17:11.658 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/35 file after the snapshot 2024-03-27T06:17:11.660 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/36 file after the snapshot 2024-03-27T06:17:11.661 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/37 file after the snapshot 2024-03-27T06:17:11.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/38 file after the snapshot 2024-03-27T06:17:11.665 INFO:tasks.workunit.client.0.smithi132.stdout: create 224/39 file after the snapshot 2024-03-27T06:17:11.667 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/.snap/snap-subdir-test 2024-03-27T06:17:11.709 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/30 file after the snapshot 2024-03-27T06:17:11.710 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/31 file after the snapshot 2024-03-27T06:17:11.711 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/32 file after the snapshot 2024-03-27T06:17:11.713 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/33 file after the snapshot 2024-03-27T06:17:11.714 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/34 file after the snapshot 2024-03-27T06:17:11.715 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/35 file after the snapshot 2024-03-27T06:17:11.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/36 file after the snapshot 2024-03-27T06:17:11.717 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/37 file after the snapshot 2024-03-27T06:17:11.719 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/38 file after the snapshot 2024-03-27T06:17:11.720 INFO:tasks.workunit.client.0.smithi132.stdout: create 225/39 file after the snapshot 2024-03-27T06:17:11.721 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/.snap/snap-subdir-test 2024-03-27T06:17:11.762 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/30 file after the snapshot 2024-03-27T06:17:11.763 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/31 file after the snapshot 2024-03-27T06:17:11.764 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/32 file after the snapshot 2024-03-27T06:17:11.765 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/33 file after the snapshot 2024-03-27T06:17:11.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/34 file after the snapshot 2024-03-27T06:17:11.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/35 file after the snapshot 2024-03-27T06:17:11.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/36 file after the snapshot 2024-03-27T06:17:11.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/37 file after the snapshot 2024-03-27T06:17:11.769 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/38 file after the snapshot 2024-03-27T06:17:11.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 226/39 file after the snapshot 2024-03-27T06:17:11.771 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/.snap/snap-subdir-test 2024-03-27T06:17:11.812 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/30 file after the snapshot 2024-03-27T06:17:11.813 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/31 file after the snapshot 2024-03-27T06:17:11.815 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/32 file after the snapshot 2024-03-27T06:17:11.816 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/33 file after the snapshot 2024-03-27T06:17:11.817 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/34 file after the snapshot 2024-03-27T06:17:11.818 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/35 file after the snapshot 2024-03-27T06:17:11.820 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/36 file after the snapshot 2024-03-27T06:17:11.821 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/37 file after the snapshot 2024-03-27T06:17:11.822 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/38 file after the snapshot 2024-03-27T06:17:11.824 INFO:tasks.workunit.client.0.smithi132.stdout: create 227/39 file after the snapshot 2024-03-27T06:17:11.825 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/.snap/snap-subdir-test 2024-03-27T06:17:11.860 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/30 file after the snapshot 2024-03-27T06:17:11.861 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/31 file after the snapshot 2024-03-27T06:17:11.863 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/32 file after the snapshot 2024-03-27T06:17:11.864 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/33 file after the snapshot 2024-03-27T06:17:11.865 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/34 file after the snapshot 2024-03-27T06:17:11.867 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/35 file after the snapshot 2024-03-27T06:17:11.868 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/36 file after the snapshot 2024-03-27T06:17:11.869 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/37 file after the snapshot 2024-03-27T06:17:11.870 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/38 file after the snapshot 2024-03-27T06:17:11.873 INFO:tasks.workunit.client.0.smithi132.stdout: create 228/39 file after the snapshot 2024-03-27T06:17:11.874 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/.snap/snap-subdir-test 2024-03-27T06:17:11.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/30 file after the snapshot 2024-03-27T06:17:11.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/31 file after the snapshot 2024-03-27T06:17:11.917 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/32 file after the snapshot 2024-03-27T06:17:11.917 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/33 file after the snapshot 2024-03-27T06:17:11.919 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/34 file after the snapshot 2024-03-27T06:17:11.920 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/35 file after the snapshot 2024-03-27T06:17:11.921 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/36 file after the snapshot 2024-03-27T06:17:11.923 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/37 file after the snapshot 2024-03-27T06:17:11.924 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/38 file after the snapshot 2024-03-27T06:17:11.925 INFO:tasks.workunit.client.0.smithi132.stdout: create 229/39 file after the snapshot 2024-03-27T06:17:11.926 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/.snap/snap-subdir-test 2024-03-27T06:17:11.967 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/30 file after the snapshot 2024-03-27T06:17:11.969 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/31 file after the snapshot 2024-03-27T06:17:11.970 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/32 file after the snapshot 2024-03-27T06:17:11.971 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/33 file after the snapshot 2024-03-27T06:17:11.973 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/34 file after the snapshot 2024-03-27T06:17:11.974 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/35 file after the snapshot 2024-03-27T06:17:11.975 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/36 file after the snapshot 2024-03-27T06:17:11.976 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/37 file after the snapshot 2024-03-27T06:17:11.978 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/38 file after the snapshot 2024-03-27T06:17:11.979 INFO:tasks.workunit.client.0.smithi132.stdout: create 230/39 file after the snapshot 2024-03-27T06:17:11.980 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/.snap/snap-subdir-test 2024-03-27T06:17:12.017 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/30 file after the snapshot 2024-03-27T06:17:12.018 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/31 file after the snapshot 2024-03-27T06:17:12.019 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/32 file after the snapshot 2024-03-27T06:17:12.021 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/33 file after the snapshot 2024-03-27T06:17:12.022 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/34 file after the snapshot 2024-03-27T06:17:12.023 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/35 file after the snapshot 2024-03-27T06:17:12.024 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/36 file after the snapshot 2024-03-27T06:17:12.026 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/37 file after the snapshot 2024-03-27T06:17:12.027 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/38 file after the snapshot 2024-03-27T06:17:12.028 INFO:tasks.workunit.client.0.smithi132.stdout: create 231/39 file after the snapshot 2024-03-27T06:17:12.029 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/.snap/snap-subdir-test 2024-03-27T06:17:12.067 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/30 file after the snapshot 2024-03-27T06:17:12.068 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/31 file after the snapshot 2024-03-27T06:17:12.070 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/32 file after the snapshot 2024-03-27T06:17:12.072 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/33 file after the snapshot 2024-03-27T06:17:12.074 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/34 file after the snapshot 2024-03-27T06:17:12.076 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/35 file after the snapshot 2024-03-27T06:17:12.077 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/36 file after the snapshot 2024-03-27T06:17:12.079 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/37 file after the snapshot 2024-03-27T06:17:12.081 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/38 file after the snapshot 2024-03-27T06:17:12.083 INFO:tasks.workunit.client.0.smithi132.stdout: create 232/39 file after the snapshot 2024-03-27T06:17:12.085 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/.snap/snap-subdir-test 2024-03-27T06:17:12.127 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/30 file after the snapshot 2024-03-27T06:17:12.129 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/31 file after the snapshot 2024-03-27T06:17:12.130 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/32 file after the snapshot 2024-03-27T06:17:12.132 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/33 file after the snapshot 2024-03-27T06:17:12.134 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/34 file after the snapshot 2024-03-27T06:17:12.135 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/35 file after the snapshot 2024-03-27T06:17:12.137 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/36 file after the snapshot 2024-03-27T06:17:12.138 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/37 file after the snapshot 2024-03-27T06:17:12.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/38 file after the snapshot 2024-03-27T06:17:12.142 INFO:tasks.workunit.client.0.smithi132.stdout: create 233/39 file after the snapshot 2024-03-27T06:17:12.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/.snap/snap-subdir-test 2024-03-27T06:17:12.182 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/30 file after the snapshot 2024-03-27T06:17:12.183 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/31 file after the snapshot 2024-03-27T06:17:12.185 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/32 file after the snapshot 2024-03-27T06:17:12.187 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/33 file after the snapshot 2024-03-27T06:17:12.189 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/34 file after the snapshot 2024-03-27T06:17:12.191 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/35 file after the snapshot 2024-03-27T06:17:12.192 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/36 file after the snapshot 2024-03-27T06:17:12.194 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/37 file after the snapshot 2024-03-27T06:17:12.196 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/38 file after the snapshot 2024-03-27T06:17:12.198 INFO:tasks.workunit.client.0.smithi132.stdout: create 234/39 file after the snapshot 2024-03-27T06:17:12.199 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/.snap/snap-subdir-test 2024-03-27T06:17:12.237 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/30 file after the snapshot 2024-03-27T06:17:12.239 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/31 file after the snapshot 2024-03-27T06:17:12.241 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/32 file after the snapshot 2024-03-27T06:17:12.242 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/33 file after the snapshot 2024-03-27T06:17:12.244 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/34 file after the snapshot 2024-03-27T06:17:12.246 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/35 file after the snapshot 2024-03-27T06:17:12.248 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/36 file after the snapshot 2024-03-27T06:17:12.249 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/37 file after the snapshot 2024-03-27T06:17:12.251 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/38 file after the snapshot 2024-03-27T06:17:12.253 INFO:tasks.workunit.client.0.smithi132.stdout: create 235/39 file after the snapshot 2024-03-27T06:17:12.254 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/.snap/snap-subdir-test 2024-03-27T06:17:12.293 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/30 file after the snapshot 2024-03-27T06:17:12.294 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/31 file after the snapshot 2024-03-27T06:17:12.296 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/32 file after the snapshot 2024-03-27T06:17:12.297 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/33 file after the snapshot 2024-03-27T06:17:12.298 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/34 file after the snapshot 2024-03-27T06:17:12.299 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/35 file after the snapshot 2024-03-27T06:17:12.301 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/36 file after the snapshot 2024-03-27T06:17:12.302 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/37 file after the snapshot 2024-03-27T06:17:12.303 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/38 file after the snapshot 2024-03-27T06:17:12.305 INFO:tasks.workunit.client.0.smithi132.stdout: create 236/39 file after the snapshot 2024-03-27T06:17:12.306 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/.snap/snap-subdir-test 2024-03-27T06:17:12.347 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/30 file after the snapshot 2024-03-27T06:17:12.348 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/31 file after the snapshot 2024-03-27T06:17:12.349 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/32 file after the snapshot 2024-03-27T06:17:12.351 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/33 file after the snapshot 2024-03-27T06:17:12.352 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/34 file after the snapshot 2024-03-27T06:17:12.353 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/35 file after the snapshot 2024-03-27T06:17:12.355 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/36 file after the snapshot 2024-03-27T06:17:12.357 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/37 file after the snapshot 2024-03-27T06:17:12.359 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/38 file after the snapshot 2024-03-27T06:17:12.361 INFO:tasks.workunit.client.0.smithi132.stdout: create 237/39 file after the snapshot 2024-03-27T06:17:12.362 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/.snap/snap-subdir-test 2024-03-27T06:17:12.401 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/30 file after the snapshot 2024-03-27T06:17:12.402 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/31 file after the snapshot 2024-03-27T06:17:12.404 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/32 file after the snapshot 2024-03-27T06:17:12.405 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/33 file after the snapshot 2024-03-27T06:17:12.406 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/34 file after the snapshot 2024-03-27T06:17:12.410 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/35 file after the snapshot 2024-03-27T06:17:12.410 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/36 file after the snapshot 2024-03-27T06:17:12.410 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/37 file after the snapshot 2024-03-27T06:17:12.412 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/38 file after the snapshot 2024-03-27T06:17:12.413 INFO:tasks.workunit.client.0.smithi132.stdout: create 238/39 file after the snapshot 2024-03-27T06:17:12.415 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/.snap/snap-subdir-test 2024-03-27T06:17:12.450 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/30 file after the snapshot 2024-03-27T06:17:12.451 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/31 file after the snapshot 2024-03-27T06:17:12.453 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/32 file after the snapshot 2024-03-27T06:17:12.454 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/33 file after the snapshot 2024-03-27T06:17:12.455 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/34 file after the snapshot 2024-03-27T06:17:12.456 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/35 file after the snapshot 2024-03-27T06:17:12.458 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/36 file after the snapshot 2024-03-27T06:17:12.459 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/37 file after the snapshot 2024-03-27T06:17:12.460 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/38 file after the snapshot 2024-03-27T06:17:12.461 INFO:tasks.workunit.client.0.smithi132.stdout: create 239/39 file after the snapshot 2024-03-27T06:17:12.463 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/.snap/snap-subdir-test 2024-03-27T06:17:12.496 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/30 file after the snapshot 2024-03-27T06:17:12.497 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/31 file after the snapshot 2024-03-27T06:17:12.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/32 file after the snapshot 2024-03-27T06:17:12.500 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/33 file after the snapshot 2024-03-27T06:17:12.501 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/34 file after the snapshot 2024-03-27T06:17:12.502 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/35 file after the snapshot 2024-03-27T06:17:12.504 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/36 file after the snapshot 2024-03-27T06:17:12.505 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/37 file after the snapshot 2024-03-27T06:17:12.506 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/38 file after the snapshot 2024-03-27T06:17:12.507 INFO:tasks.workunit.client.0.smithi132.stdout: create 240/39 file after the snapshot 2024-03-27T06:17:12.508 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/.snap/snap-subdir-test 2024-03-27T06:17:12.544 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/30 file after the snapshot 2024-03-27T06:17:12.546 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/31 file after the snapshot 2024-03-27T06:17:12.548 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/32 file after the snapshot 2024-03-27T06:17:12.549 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/33 file after the snapshot 2024-03-27T06:17:12.551 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/34 file after the snapshot 2024-03-27T06:17:12.553 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/35 file after the snapshot 2024-03-27T06:17:12.555 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/36 file after the snapshot 2024-03-27T06:17:12.556 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/37 file after the snapshot 2024-03-27T06:17:12.558 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/38 file after the snapshot 2024-03-27T06:17:12.560 INFO:tasks.workunit.client.0.smithi132.stdout: create 241/39 file after the snapshot 2024-03-27T06:17:12.562 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/.snap/snap-subdir-test 2024-03-27T06:17:12.601 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/30 file after the snapshot 2024-03-27T06:17:12.602 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/31 file after the snapshot 2024-03-27T06:17:12.603 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/32 file after the snapshot 2024-03-27T06:17:12.604 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/33 file after the snapshot 2024-03-27T06:17:12.605 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/34 file after the snapshot 2024-03-27T06:17:12.606 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/35 file after the snapshot 2024-03-27T06:17:12.607 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/36 file after the snapshot 2024-03-27T06:17:12.609 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/37 file after the snapshot 2024-03-27T06:17:12.610 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/38 file after the snapshot 2024-03-27T06:17:12.611 INFO:tasks.workunit.client.0.smithi132.stdout: create 242/39 file after the snapshot 2024-03-27T06:17:12.613 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/.snap/snap-subdir-test 2024-03-27T06:17:12.655 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/30 file after the snapshot 2024-03-27T06:17:12.657 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/31 file after the snapshot 2024-03-27T06:17:12.658 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/32 file after the snapshot 2024-03-27T06:17:12.660 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/33 file after the snapshot 2024-03-27T06:17:12.662 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/34 file after the snapshot 2024-03-27T06:17:12.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/35 file after the snapshot 2024-03-27T06:17:12.665 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/36 file after the snapshot 2024-03-27T06:17:12.667 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/37 file after the snapshot 2024-03-27T06:17:12.669 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/38 file after the snapshot 2024-03-27T06:17:12.671 INFO:tasks.workunit.client.0.smithi132.stdout: create 243/39 file after the snapshot 2024-03-27T06:17:12.673 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/.snap/snap-subdir-test 2024-03-27T06:17:12.711 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/30 file after the snapshot 2024-03-27T06:17:12.713 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/31 file after the snapshot 2024-03-27T06:17:12.714 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/32 file after the snapshot 2024-03-27T06:17:12.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/33 file after the snapshot 2024-03-27T06:17:12.718 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/34 file after the snapshot 2024-03-27T06:17:12.719 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/35 file after the snapshot 2024-03-27T06:17:12.721 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/36 file after the snapshot 2024-03-27T06:17:12.723 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/37 file after the snapshot 2024-03-27T06:17:12.725 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/38 file after the snapshot 2024-03-27T06:17:12.728 INFO:tasks.workunit.client.0.smithi132.stdout: create 244/39 file after the snapshot 2024-03-27T06:17:12.729 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/.snap/snap-subdir-test 2024-03-27T06:17:12.764 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/30 file after the snapshot 2024-03-27T06:17:12.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/31 file after the snapshot 2024-03-27T06:17:12.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/32 file after the snapshot 2024-03-27T06:17:12.769 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/33 file after the snapshot 2024-03-27T06:17:12.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/34 file after the snapshot 2024-03-27T06:17:12.772 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/35 file after the snapshot 2024-03-27T06:17:12.774 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/36 file after the snapshot 2024-03-27T06:17:12.776 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/37 file after the snapshot 2024-03-27T06:17:12.778 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/38 file after the snapshot 2024-03-27T06:17:12.779 INFO:tasks.workunit.client.0.smithi132.stdout: create 245/39 file after the snapshot 2024-03-27T06:17:12.781 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/.snap/snap-subdir-test 2024-03-27T06:17:12.813 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/30 file after the snapshot 2024-03-27T06:17:12.815 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/31 file after the snapshot 2024-03-27T06:17:12.816 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/32 file after the snapshot 2024-03-27T06:17:12.818 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/33 file after the snapshot 2024-03-27T06:17:12.820 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/34 file after the snapshot 2024-03-27T06:17:12.821 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/35 file after the snapshot 2024-03-27T06:17:12.823 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/36 file after the snapshot 2024-03-27T06:17:12.825 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/37 file after the snapshot 2024-03-27T06:17:12.826 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/38 file after the snapshot 2024-03-27T06:17:12.828 INFO:tasks.workunit.client.0.smithi132.stdout: create 246/39 file after the snapshot 2024-03-27T06:17:12.829 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/.snap/snap-subdir-test 2024-03-27T06:17:12.864 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/30 file after the snapshot 2024-03-27T06:17:12.866 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/31 file after the snapshot 2024-03-27T06:17:12.868 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/32 file after the snapshot 2024-03-27T06:17:12.869 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/33 file after the snapshot 2024-03-27T06:17:12.871 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/34 file after the snapshot 2024-03-27T06:17:12.873 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/35 file after the snapshot 2024-03-27T06:17:12.875 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/36 file after the snapshot 2024-03-27T06:17:12.877 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/37 file after the snapshot 2024-03-27T06:17:12.879 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/38 file after the snapshot 2024-03-27T06:17:12.881 INFO:tasks.workunit.client.0.smithi132.stdout: create 247/39 file after the snapshot 2024-03-27T06:17:12.882 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/.snap/snap-subdir-test 2024-03-27T06:17:12.913 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/30 file after the snapshot 2024-03-27T06:17:12.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/31 file after the snapshot 2024-03-27T06:17:12.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/32 file after the snapshot 2024-03-27T06:17:12.916 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/33 file after the snapshot 2024-03-27T06:17:12.917 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/34 file after the snapshot 2024-03-27T06:17:12.918 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/35 file after the snapshot 2024-03-27T06:17:12.919 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/36 file after the snapshot 2024-03-27T06:17:12.921 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/37 file after the snapshot 2024-03-27T06:17:12.921 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/38 file after the snapshot 2024-03-27T06:17:12.922 INFO:tasks.workunit.client.0.smithi132.stdout: create 248/39 file after the snapshot 2024-03-27T06:17:12.923 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/.snap/snap-subdir-test 2024-03-27T06:17:12.951 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/30 file after the snapshot 2024-03-27T06:17:12.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/31 file after the snapshot 2024-03-27T06:17:12.954 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/32 file after the snapshot 2024-03-27T06:17:12.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/33 file after the snapshot 2024-03-27T06:17:12.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/34 file after the snapshot 2024-03-27T06:17:12.960 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/35 file after the snapshot 2024-03-27T06:17:12.962 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/36 file after the snapshot 2024-03-27T06:17:12.963 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/37 file after the snapshot 2024-03-27T06:17:12.965 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/38 file after the snapshot 2024-03-27T06:17:12.967 INFO:tasks.workunit.client.0.smithi132.stdout: create 249/39 file after the snapshot 2024-03-27T06:17:12.969 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/.snap/snap-subdir-test 2024-03-27T06:17:13.002 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/30 file after the snapshot 2024-03-27T06:17:13.004 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/31 file after the snapshot 2024-03-27T06:17:13.006 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/32 file after the snapshot 2024-03-27T06:17:13.008 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/33 file after the snapshot 2024-03-27T06:17:13.009 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/34 file after the snapshot 2024-03-27T06:17:13.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/35 file after the snapshot 2024-03-27T06:17:13.013 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/36 file after the snapshot 2024-03-27T06:17:13.014 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/37 file after the snapshot 2024-03-27T06:17:13.016 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/38 file after the snapshot 2024-03-27T06:17:13.018 INFO:tasks.workunit.client.0.smithi132.stdout: create 250/39 file after the snapshot 2024-03-27T06:17:13.021 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/.snap/snap-subdir-test 2024-03-27T06:17:13.052 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/30 file after the snapshot 2024-03-27T06:17:13.053 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/31 file after the snapshot 2024-03-27T06:17:13.054 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/32 file after the snapshot 2024-03-27T06:17:13.055 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/33 file after the snapshot 2024-03-27T06:17:13.057 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/34 file after the snapshot 2024-03-27T06:17:13.058 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/35 file after the snapshot 2024-03-27T06:17:13.059 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/36 file after the snapshot 2024-03-27T06:17:13.060 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/37 file after the snapshot 2024-03-27T06:17:13.062 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/38 file after the snapshot 2024-03-27T06:17:13.063 INFO:tasks.workunit.client.0.smithi132.stdout: create 251/39 file after the snapshot 2024-03-27T06:17:13.064 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/.snap/snap-subdir-test 2024-03-27T06:17:13.097 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/30 file after the snapshot 2024-03-27T06:17:13.098 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/31 file after the snapshot 2024-03-27T06:17:13.099 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/32 file after the snapshot 2024-03-27T06:17:13.100 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/33 file after the snapshot 2024-03-27T06:17:13.101 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/34 file after the snapshot 2024-03-27T06:17:13.102 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/35 file after the snapshot 2024-03-27T06:17:13.104 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/36 file after the snapshot 2024-03-27T06:17:13.104 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/37 file after the snapshot 2024-03-27T06:17:13.105 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/38 file after the snapshot 2024-03-27T06:17:13.106 INFO:tasks.workunit.client.0.smithi132.stdout: create 252/39 file after the snapshot 2024-03-27T06:17:13.107 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/.snap/snap-subdir-test 2024-03-27T06:17:13.138 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/30 file after the snapshot 2024-03-27T06:17:13.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/31 file after the snapshot 2024-03-27T06:17:13.142 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/32 file after the snapshot 2024-03-27T06:17:13.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/33 file after the snapshot 2024-03-27T06:17:13.146 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/34 file after the snapshot 2024-03-27T06:17:13.148 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/35 file after the snapshot 2024-03-27T06:17:13.149 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/36 file after the snapshot 2024-03-27T06:17:13.151 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/37 file after the snapshot 2024-03-27T06:17:13.153 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/38 file after the snapshot 2024-03-27T06:17:13.155 INFO:tasks.workunit.client.0.smithi132.stdout: create 253/39 file after the snapshot 2024-03-27T06:17:13.156 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/.snap/snap-subdir-test 2024-03-27T06:17:13.188 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/30 file after the snapshot 2024-03-27T06:17:13.190 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/31 file after the snapshot 2024-03-27T06:17:13.191 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/32 file after the snapshot 2024-03-27T06:17:13.193 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/33 file after the snapshot 2024-03-27T06:17:13.195 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/34 file after the snapshot 2024-03-27T06:17:13.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/35 file after the snapshot 2024-03-27T06:17:13.199 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/36 file after the snapshot 2024-03-27T06:17:13.201 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/37 file after the snapshot 2024-03-27T06:17:13.203 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/38 file after the snapshot 2024-03-27T06:17:13.204 INFO:tasks.workunit.client.0.smithi132.stdout: create 254/39 file after the snapshot 2024-03-27T06:17:13.207 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/.snap/snap-subdir-test 2024-03-27T06:17:13.239 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/30 file after the snapshot 2024-03-27T06:17:13.241 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/31 file after the snapshot 2024-03-27T06:17:13.243 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/32 file after the snapshot 2024-03-27T06:17:13.244 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/33 file after the snapshot 2024-03-27T06:17:13.246 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/34 file after the snapshot 2024-03-27T06:17:13.248 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/35 file after the snapshot 2024-03-27T06:17:13.250 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/36 file after the snapshot 2024-03-27T06:17:13.252 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/37 file after the snapshot 2024-03-27T06:17:13.254 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/38 file after the snapshot 2024-03-27T06:17:13.256 INFO:tasks.workunit.client.0.smithi132.stdout: create 255/39 file after the snapshot 2024-03-27T06:17:13.257 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/.snap/snap-subdir-test 2024-03-27T06:17:13.292 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/30 file after the snapshot 2024-03-27T06:17:13.293 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/31 file after the snapshot 2024-03-27T06:17:13.295 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/32 file after the snapshot 2024-03-27T06:17:13.296 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/33 file after the snapshot 2024-03-27T06:17:13.297 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/34 file after the snapshot 2024-03-27T06:17:13.299 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/35 file after the snapshot 2024-03-27T06:17:13.300 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/36 file after the snapshot 2024-03-27T06:17:13.301 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/37 file after the snapshot 2024-03-27T06:17:13.302 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/38 file after the snapshot 2024-03-27T06:17:13.304 INFO:tasks.workunit.client.0.smithi132.stdout: create 256/39 file after the snapshot 2024-03-27T06:17:13.305 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/.snap/snap-subdir-test 2024-03-27T06:17:13.336 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/30 file after the snapshot 2024-03-27T06:17:13.338 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/31 file after the snapshot 2024-03-27T06:17:13.340 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/32 file after the snapshot 2024-03-27T06:17:13.342 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/33 file after the snapshot 2024-03-27T06:17:13.343 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/34 file after the snapshot 2024-03-27T06:17:13.345 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/35 file after the snapshot 2024-03-27T06:17:13.347 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/36 file after the snapshot 2024-03-27T06:17:13.349 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/37 file after the snapshot 2024-03-27T06:17:13.351 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/38 file after the snapshot 2024-03-27T06:17:13.353 INFO:tasks.workunit.client.0.smithi132.stdout: create 257/39 file after the snapshot 2024-03-27T06:17:13.355 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/.snap/snap-subdir-test 2024-03-27T06:17:13.384 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/30 file after the snapshot 2024-03-27T06:17:13.386 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/31 file after the snapshot 2024-03-27T06:17:13.388 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/32 file after the snapshot 2024-03-27T06:17:13.390 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/33 file after the snapshot 2024-03-27T06:17:13.391 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/34 file after the snapshot 2024-03-27T06:17:13.393 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/35 file after the snapshot 2024-03-27T06:17:13.395 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/36 file after the snapshot 2024-03-27T06:17:13.397 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/37 file after the snapshot 2024-03-27T06:17:13.399 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/38 file after the snapshot 2024-03-27T06:17:13.401 INFO:tasks.workunit.client.0.smithi132.stdout: create 258/39 file after the snapshot 2024-03-27T06:17:13.403 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/.snap/snap-subdir-test 2024-03-27T06:17:13.438 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/30 file after the snapshot 2024-03-27T06:17:13.440 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/31 file after the snapshot 2024-03-27T06:17:13.441 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/32 file after the snapshot 2024-03-27T06:17:13.442 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/33 file after the snapshot 2024-03-27T06:17:13.443 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/34 file after the snapshot 2024-03-27T06:17:13.445 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/35 file after the snapshot 2024-03-27T06:17:13.446 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/36 file after the snapshot 2024-03-27T06:17:13.447 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/37 file after the snapshot 2024-03-27T06:17:13.448 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/38 file after the snapshot 2024-03-27T06:17:13.450 INFO:tasks.workunit.client.0.smithi132.stdout: create 259/39 file after the snapshot 2024-03-27T06:17:13.451 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/.snap/snap-subdir-test 2024-03-27T06:17:13.484 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/30 file after the snapshot 2024-03-27T06:17:13.486 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/31 file after the snapshot 2024-03-27T06:17:13.489 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/32 file after the snapshot 2024-03-27T06:17:13.489 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/33 file after the snapshot 2024-03-27T06:17:13.491 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/34 file after the snapshot 2024-03-27T06:17:13.493 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/35 file after the snapshot 2024-03-27T06:17:13.495 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/36 file after the snapshot 2024-03-27T06:17:13.497 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/37 file after the snapshot 2024-03-27T06:17:13.499 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/38 file after the snapshot 2024-03-27T06:17:13.500 INFO:tasks.workunit.client.0.smithi132.stdout: create 260/39 file after the snapshot 2024-03-27T06:17:13.502 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/.snap/snap-subdir-test 2024-03-27T06:17:13.533 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/30 file after the snapshot 2024-03-27T06:17:13.534 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/31 file after the snapshot 2024-03-27T06:17:13.536 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/32 file after the snapshot 2024-03-27T06:17:13.537 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/33 file after the snapshot 2024-03-27T06:17:13.538 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/34 file after the snapshot 2024-03-27T06:17:13.540 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/35 file after the snapshot 2024-03-27T06:17:13.541 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/36 file after the snapshot 2024-03-27T06:17:13.543 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/37 file after the snapshot 2024-03-27T06:17:13.544 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/38 file after the snapshot 2024-03-27T06:17:13.545 INFO:tasks.workunit.client.0.smithi132.stdout: create 261/39 file after the snapshot 2024-03-27T06:17:13.546 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/.snap/snap-subdir-test 2024-03-27T06:17:13.581 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/30 file after the snapshot 2024-03-27T06:17:13.582 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/31 file after the snapshot 2024-03-27T06:17:13.584 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/32 file after the snapshot 2024-03-27T06:17:13.585 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/33 file after the snapshot 2024-03-27T06:17:13.586 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/34 file after the snapshot 2024-03-27T06:17:13.588 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/35 file after the snapshot 2024-03-27T06:17:13.590 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/36 file after the snapshot 2024-03-27T06:17:13.592 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/37 file after the snapshot 2024-03-27T06:17:13.593 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/38 file after the snapshot 2024-03-27T06:17:13.594 INFO:tasks.workunit.client.0.smithi132.stdout: create 262/39 file after the snapshot 2024-03-27T06:17:13.595 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/.snap/snap-subdir-test 2024-03-27T06:17:13.628 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/30 file after the snapshot 2024-03-27T06:17:13.629 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/31 file after the snapshot 2024-03-27T06:17:13.631 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/32 file after the snapshot 2024-03-27T06:17:13.633 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/33 file after the snapshot 2024-03-27T06:17:13.634 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/34 file after the snapshot 2024-03-27T06:17:13.636 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/35 file after the snapshot 2024-03-27T06:17:13.638 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/36 file after the snapshot 2024-03-27T06:17:13.640 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/37 file after the snapshot 2024-03-27T06:17:13.641 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/38 file after the snapshot 2024-03-27T06:17:13.643 INFO:tasks.workunit.client.0.smithi132.stdout: create 263/39 file after the snapshot 2024-03-27T06:17:13.644 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/.snap/snap-subdir-test 2024-03-27T06:17:13.678 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/30 file after the snapshot 2024-03-27T06:17:13.679 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/31 file after the snapshot 2024-03-27T06:17:13.681 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/32 file after the snapshot 2024-03-27T06:17:13.682 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/33 file after the snapshot 2024-03-27T06:17:13.683 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/34 file after the snapshot 2024-03-27T06:17:13.684 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/35 file after the snapshot 2024-03-27T06:17:13.686 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/36 file after the snapshot 2024-03-27T06:17:13.687 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/37 file after the snapshot 2024-03-27T06:17:13.688 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/38 file after the snapshot 2024-03-27T06:17:13.689 INFO:tasks.workunit.client.0.smithi132.stdout: create 264/39 file after the snapshot 2024-03-27T06:17:13.691 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/.snap/snap-subdir-test 2024-03-27T06:17:13.720 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/30 file after the snapshot 2024-03-27T06:17:13.722 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/31 file after the snapshot 2024-03-27T06:17:13.723 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/32 file after the snapshot 2024-03-27T06:17:13.725 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/33 file after the snapshot 2024-03-27T06:17:13.726 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/34 file after the snapshot 2024-03-27T06:17:13.728 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/35 file after the snapshot 2024-03-27T06:17:13.729 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/36 file after the snapshot 2024-03-27T06:17:13.731 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/37 file after the snapshot 2024-03-27T06:17:13.732 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/38 file after the snapshot 2024-03-27T06:17:13.733 INFO:tasks.workunit.client.0.smithi132.stdout: create 265/39 file after the snapshot 2024-03-27T06:17:13.735 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/.snap/snap-subdir-test 2024-03-27T06:17:13.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/30 file after the snapshot 2024-03-27T06:17:13.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/31 file after the snapshot 2024-03-27T06:17:13.769 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/32 file after the snapshot 2024-03-27T06:17:13.770 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/33 file after the snapshot 2024-03-27T06:17:13.772 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/34 file after the snapshot 2024-03-27T06:17:13.773 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/35 file after the snapshot 2024-03-27T06:17:13.775 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/36 file after the snapshot 2024-03-27T06:17:13.776 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/37 file after the snapshot 2024-03-27T06:17:13.778 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/38 file after the snapshot 2024-03-27T06:17:13.779 INFO:tasks.workunit.client.0.smithi132.stdout: create 266/39 file after the snapshot 2024-03-27T06:17:13.781 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/.snap/snap-subdir-test 2024-03-27T06:17:13.813 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/30 file after the snapshot 2024-03-27T06:17:13.814 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/31 file after the snapshot 2024-03-27T06:17:13.815 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/32 file after the snapshot 2024-03-27T06:17:13.817 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/33 file after the snapshot 2024-03-27T06:17:13.818 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/34 file after the snapshot 2024-03-27T06:17:13.819 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/35 file after the snapshot 2024-03-27T06:17:13.821 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/36 file after the snapshot 2024-03-27T06:17:13.822 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/37 file after the snapshot 2024-03-27T06:17:13.823 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/38 file after the snapshot 2024-03-27T06:17:13.824 INFO:tasks.workunit.client.0.smithi132.stdout: create 267/39 file after the snapshot 2024-03-27T06:17:13.826 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/.snap/snap-subdir-test 2024-03-27T06:17:13.857 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/30 file after the snapshot 2024-03-27T06:17:13.858 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/31 file after the snapshot 2024-03-27T06:17:13.860 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/32 file after the snapshot 2024-03-27T06:17:13.862 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/33 file after the snapshot 2024-03-27T06:17:13.864 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/34 file after the snapshot 2024-03-27T06:17:13.866 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/35 file after the snapshot 2024-03-27T06:17:13.868 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/36 file after the snapshot 2024-03-27T06:17:13.870 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/37 file after the snapshot 2024-03-27T06:17:13.871 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/38 file after the snapshot 2024-03-27T06:17:13.873 INFO:tasks.workunit.client.0.smithi132.stdout: create 268/39 file after the snapshot 2024-03-27T06:17:13.876 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/.snap/snap-subdir-test 2024-03-27T06:17:13.906 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/30 file after the snapshot 2024-03-27T06:17:13.907 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/31 file after the snapshot 2024-03-27T06:17:13.908 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/32 file after the snapshot 2024-03-27T06:17:13.909 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/33 file after the snapshot 2024-03-27T06:17:13.910 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/34 file after the snapshot 2024-03-27T06:17:13.911 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/35 file after the snapshot 2024-03-27T06:17:13.912 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/36 file after the snapshot 2024-03-27T06:17:13.913 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/37 file after the snapshot 2024-03-27T06:17:13.914 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/38 file after the snapshot 2024-03-27T06:17:13.915 INFO:tasks.workunit.client.0.smithi132.stdout: create 269/39 file after the snapshot 2024-03-27T06:17:13.916 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/.snap/snap-subdir-test 2024-03-27T06:17:13.947 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/30 file after the snapshot 2024-03-27T06:17:13.948 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/31 file after the snapshot 2024-03-27T06:17:13.949 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/32 file after the snapshot 2024-03-27T06:17:13.951 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/33 file after the snapshot 2024-03-27T06:17:13.952 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/34 file after the snapshot 2024-03-27T06:17:13.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/35 file after the snapshot 2024-03-27T06:17:13.955 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/36 file after the snapshot 2024-03-27T06:17:13.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/37 file after the snapshot 2024-03-27T06:17:13.958 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/38 file after the snapshot 2024-03-27T06:17:13.959 INFO:tasks.workunit.client.0.smithi132.stdout: create 270/39 file after the snapshot 2024-03-27T06:17:13.960 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/.snap/snap-subdir-test 2024-03-27T06:17:13.992 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/30 file after the snapshot 2024-03-27T06:17:13.993 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/31 file after the snapshot 2024-03-27T06:17:13.995 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/32 file after the snapshot 2024-03-27T06:17:13.997 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/33 file after the snapshot 2024-03-27T06:17:13.998 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/34 file after the snapshot 2024-03-27T06:17:13.999 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/35 file after the snapshot 2024-03-27T06:17:14.000 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/36 file after the snapshot 2024-03-27T06:17:14.002 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/37 file after the snapshot 2024-03-27T06:17:14.003 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/38 file after the snapshot 2024-03-27T06:17:14.011 INFO:tasks.workunit.client.0.smithi132.stdout: create 271/39 file after the snapshot 2024-03-27T06:17:14.013 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/.snap/snap-subdir-test 2024-03-27T06:17:14.043 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/30 file after the snapshot 2024-03-27T06:17:14.044 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/31 file after the snapshot 2024-03-27T06:17:14.045 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/32 file after the snapshot 2024-03-27T06:17:14.046 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/33 file after the snapshot 2024-03-27T06:17:14.047 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/34 file after the snapshot 2024-03-27T06:17:14.048 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/35 file after the snapshot 2024-03-27T06:17:14.049 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/36 file after the snapshot 2024-03-27T06:17:14.050 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/37 file after the snapshot 2024-03-27T06:17:14.051 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/38 file after the snapshot 2024-03-27T06:17:14.052 INFO:tasks.workunit.client.0.smithi132.stdout: create 272/39 file after the snapshot 2024-03-27T06:17:14.053 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/.snap/snap-subdir-test 2024-03-27T06:17:14.086 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/30 file after the snapshot 2024-03-27T06:17:14.088 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/31 file after the snapshot 2024-03-27T06:17:14.089 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/32 file after the snapshot 2024-03-27T06:17:14.091 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/33 file after the snapshot 2024-03-27T06:17:14.092 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/34 file after the snapshot 2024-03-27T06:17:14.094 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/35 file after the snapshot 2024-03-27T06:17:14.095 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/36 file after the snapshot 2024-03-27T06:17:14.097 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/37 file after the snapshot 2024-03-27T06:17:14.098 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/38 file after the snapshot 2024-03-27T06:17:14.100 INFO:tasks.workunit.client.0.smithi132.stdout: create 273/39 file after the snapshot 2024-03-27T06:17:14.101 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/.snap/snap-subdir-test 2024-03-27T06:17:14.136 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/30 file after the snapshot 2024-03-27T06:17:14.137 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/31 file after the snapshot 2024-03-27T06:17:14.139 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/32 file after the snapshot 2024-03-27T06:17:14.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/33 file after the snapshot 2024-03-27T06:17:14.142 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/34 file after the snapshot 2024-03-27T06:17:14.143 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/35 file after the snapshot 2024-03-27T06:17:14.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/36 file after the snapshot 2024-03-27T06:17:14.146 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/37 file after the snapshot 2024-03-27T06:17:14.147 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/38 file after the snapshot 2024-03-27T06:17:14.149 INFO:tasks.workunit.client.0.smithi132.stdout: create 274/39 file after the snapshot 2024-03-27T06:17:14.150 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/.snap/snap-subdir-test 2024-03-27T06:17:14.182 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/30 file after the snapshot 2024-03-27T06:17:14.184 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/31 file after the snapshot 2024-03-27T06:17:14.185 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/32 file after the snapshot 2024-03-27T06:17:14.187 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/33 file after the snapshot 2024-03-27T06:17:14.188 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/34 file after the snapshot 2024-03-27T06:17:14.190 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/35 file after the snapshot 2024-03-27T06:17:14.192 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/36 file after the snapshot 2024-03-27T06:17:14.193 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/37 file after the snapshot 2024-03-27T06:17:14.195 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/38 file after the snapshot 2024-03-27T06:17:14.197 INFO:tasks.workunit.client.0.smithi132.stdout: create 275/39 file after the snapshot 2024-03-27T06:17:14.198 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/.snap/snap-subdir-test 2024-03-27T06:17:14.230 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/30 file after the snapshot 2024-03-27T06:17:14.231 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/31 file after the snapshot 2024-03-27T06:17:14.233 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/32 file after the snapshot 2024-03-27T06:17:14.235 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/33 file after the snapshot 2024-03-27T06:17:14.236 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/34 file after the snapshot 2024-03-27T06:17:14.238 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/35 file after the snapshot 2024-03-27T06:17:14.239 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/36 file after the snapshot 2024-03-27T06:17:14.240 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/37 file after the snapshot 2024-03-27T06:17:14.241 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/38 file after the snapshot 2024-03-27T06:17:14.243 INFO:tasks.workunit.client.0.smithi132.stdout: create 276/39 file after the snapshot 2024-03-27T06:17:14.244 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/.snap/snap-subdir-test 2024-03-27T06:17:14.275 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/30 file after the snapshot 2024-03-27T06:17:14.277 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/31 file after the snapshot 2024-03-27T06:17:14.278 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/32 file after the snapshot 2024-03-27T06:17:14.279 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/33 file after the snapshot 2024-03-27T06:17:14.281 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/34 file after the snapshot 2024-03-27T06:17:14.282 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/35 file after the snapshot 2024-03-27T06:17:14.283 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/36 file after the snapshot 2024-03-27T06:17:14.285 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/37 file after the snapshot 2024-03-27T06:17:14.286 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/38 file after the snapshot 2024-03-27T06:17:14.287 INFO:tasks.workunit.client.0.smithi132.stdout: create 277/39 file after the snapshot 2024-03-27T06:17:14.289 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/.snap/snap-subdir-test 2024-03-27T06:17:14.319 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/30 file after the snapshot 2024-03-27T06:17:14.321 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/31 file after the snapshot 2024-03-27T06:17:14.322 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/32 file after the snapshot 2024-03-27T06:17:14.323 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/33 file after the snapshot 2024-03-27T06:17:14.325 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/34 file after the snapshot 2024-03-27T06:17:14.326 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/35 file after the snapshot 2024-03-27T06:17:14.327 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/36 file after the snapshot 2024-03-27T06:17:14.328 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/37 file after the snapshot 2024-03-27T06:17:14.330 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/38 file after the snapshot 2024-03-27T06:17:14.331 INFO:tasks.workunit.client.0.smithi132.stdout: create 278/39 file after the snapshot 2024-03-27T06:17:14.332 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/.snap/snap-subdir-test 2024-03-27T06:17:14.364 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/30 file after the snapshot 2024-03-27T06:17:14.366 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/31 file after the snapshot 2024-03-27T06:17:14.368 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/32 file after the snapshot 2024-03-27T06:17:14.369 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/33 file after the snapshot 2024-03-27T06:17:14.371 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/34 file after the snapshot 2024-03-27T06:17:14.373 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/35 file after the snapshot 2024-03-27T06:17:14.374 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/36 file after the snapshot 2024-03-27T06:17:14.376 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/37 file after the snapshot 2024-03-27T06:17:14.378 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/38 file after the snapshot 2024-03-27T06:17:14.380 INFO:tasks.workunit.client.0.smithi132.stdout: create 279/39 file after the snapshot 2024-03-27T06:17:14.381 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/.snap/snap-subdir-test 2024-03-27T06:17:14.417 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/30 file after the snapshot 2024-03-27T06:17:14.418 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/31 file after the snapshot 2024-03-27T06:17:14.419 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/32 file after the snapshot 2024-03-27T06:17:14.420 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/33 file after the snapshot 2024-03-27T06:17:14.422 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/34 file after the snapshot 2024-03-27T06:17:14.423 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/35 file after the snapshot 2024-03-27T06:17:14.424 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/36 file after the snapshot 2024-03-27T06:17:14.425 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/37 file after the snapshot 2024-03-27T06:17:14.426 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/38 file after the snapshot 2024-03-27T06:17:14.428 INFO:tasks.workunit.client.0.smithi132.stdout: create 280/39 file after the snapshot 2024-03-27T06:17:14.429 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/.snap/snap-subdir-test 2024-03-27T06:17:14.469 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/30 file after the snapshot 2024-03-27T06:17:14.471 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/31 file after the snapshot 2024-03-27T06:17:14.472 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/32 file after the snapshot 2024-03-27T06:17:14.473 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/33 file after the snapshot 2024-03-27T06:17:14.474 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/34 file after the snapshot 2024-03-27T06:17:14.476 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/35 file after the snapshot 2024-03-27T06:17:14.477 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/36 file after the snapshot 2024-03-27T06:17:14.478 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/37 file after the snapshot 2024-03-27T06:17:14.480 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/38 file after the snapshot 2024-03-27T06:17:14.481 INFO:tasks.workunit.client.0.smithi132.stdout: create 281/39 file after the snapshot 2024-03-27T06:17:14.483 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/.snap/snap-subdir-test 2024-03-27T06:17:14.516 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/30 file after the snapshot 2024-03-27T06:17:14.518 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/31 file after the snapshot 2024-03-27T06:17:14.520 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/32 file after the snapshot 2024-03-27T06:17:14.521 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/33 file after the snapshot 2024-03-27T06:17:14.522 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/34 file after the snapshot 2024-03-27T06:17:14.524 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/35 file after the snapshot 2024-03-27T06:17:14.525 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/36 file after the snapshot 2024-03-27T06:17:14.527 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/37 file after the snapshot 2024-03-27T06:17:14.528 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/38 file after the snapshot 2024-03-27T06:17:14.529 INFO:tasks.workunit.client.0.smithi132.stdout: create 282/39 file after the snapshot 2024-03-27T06:17:14.531 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/.snap/snap-subdir-test 2024-03-27T06:17:14.570 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/30 file after the snapshot 2024-03-27T06:17:14.572 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/31 file after the snapshot 2024-03-27T06:17:14.574 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/32 file after the snapshot 2024-03-27T06:17:14.575 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/33 file after the snapshot 2024-03-27T06:17:14.577 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/34 file after the snapshot 2024-03-27T06:17:14.579 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/35 file after the snapshot 2024-03-27T06:17:14.580 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/36 file after the snapshot 2024-03-27T06:17:14.582 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/37 file after the snapshot 2024-03-27T06:17:14.584 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/38 file after the snapshot 2024-03-27T06:17:14.585 INFO:tasks.workunit.client.0.smithi132.stdout: create 283/39 file after the snapshot 2024-03-27T06:17:14.587 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/.snap/snap-subdir-test 2024-03-27T06:17:14.625 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/30 file after the snapshot 2024-03-27T06:17:14.626 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/31 file after the snapshot 2024-03-27T06:17:14.627 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/32 file after the snapshot 2024-03-27T06:17:14.628 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/33 file after the snapshot 2024-03-27T06:17:14.630 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/34 file after the snapshot 2024-03-27T06:17:14.632 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/35 file after the snapshot 2024-03-27T06:17:14.634 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/36 file after the snapshot 2024-03-27T06:17:14.636 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/37 file after the snapshot 2024-03-27T06:17:14.637 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/38 file after the snapshot 2024-03-27T06:17:14.639 INFO:tasks.workunit.client.0.smithi132.stdout: create 284/39 file after the snapshot 2024-03-27T06:17:14.641 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/.snap/snap-subdir-test 2024-03-27T06:17:14.674 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/30 file after the snapshot 2024-03-27T06:17:14.676 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/31 file after the snapshot 2024-03-27T06:17:14.677 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/32 file after the snapshot 2024-03-27T06:17:14.679 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/33 file after the snapshot 2024-03-27T06:17:14.681 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/34 file after the snapshot 2024-03-27T06:17:14.682 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/35 file after the snapshot 2024-03-27T06:17:14.684 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/36 file after the snapshot 2024-03-27T06:17:14.685 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/37 file after the snapshot 2024-03-27T06:17:14.687 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/38 file after the snapshot 2024-03-27T06:17:14.689 INFO:tasks.workunit.client.0.smithi132.stdout: create 285/39 file after the snapshot 2024-03-27T06:17:14.690 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/.snap/snap-subdir-test 2024-03-27T06:17:14.730 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/30 file after the snapshot 2024-03-27T06:17:14.732 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/31 file after the snapshot 2024-03-27T06:17:14.733 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/32 file after the snapshot 2024-03-27T06:17:14.735 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/33 file after the snapshot 2024-03-27T06:17:14.737 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/34 file after the snapshot 2024-03-27T06:17:14.738 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/35 file after the snapshot 2024-03-27T06:17:14.740 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/36 file after the snapshot 2024-03-27T06:17:14.742 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/37 file after the snapshot 2024-03-27T06:17:14.744 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/38 file after the snapshot 2024-03-27T06:17:14.746 INFO:tasks.workunit.client.0.smithi132.stdout: create 286/39 file after the snapshot 2024-03-27T06:17:14.747 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/.snap/snap-subdir-test 2024-03-27T06:17:14.780 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/30 file after the snapshot 2024-03-27T06:17:14.782 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/31 file after the snapshot 2024-03-27T06:17:14.783 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/32 file after the snapshot 2024-03-27T06:17:14.784 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/33 file after the snapshot 2024-03-27T06:17:14.785 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/34 file after the snapshot 2024-03-27T06:17:14.787 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/35 file after the snapshot 2024-03-27T06:17:14.788 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/36 file after the snapshot 2024-03-27T06:17:14.789 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/37 file after the snapshot 2024-03-27T06:17:14.791 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/38 file after the snapshot 2024-03-27T06:17:14.792 INFO:tasks.workunit.client.0.smithi132.stdout: create 287/39 file after the snapshot 2024-03-27T06:17:14.793 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/.snap/snap-subdir-test 2024-03-27T06:17:14.828 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/30 file after the snapshot 2024-03-27T06:17:14.829 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/31 file after the snapshot 2024-03-27T06:17:14.831 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/32 file after the snapshot 2024-03-27T06:17:14.833 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/33 file after the snapshot 2024-03-27T06:17:14.834 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/34 file after the snapshot 2024-03-27T06:17:14.836 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/35 file after the snapshot 2024-03-27T06:17:14.838 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/36 file after the snapshot 2024-03-27T06:17:14.840 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/37 file after the snapshot 2024-03-27T06:17:14.842 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/38 file after the snapshot 2024-03-27T06:17:14.843 INFO:tasks.workunit.client.0.smithi132.stdout: create 288/39 file after the snapshot 2024-03-27T06:17:14.845 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/.snap/snap-subdir-test 2024-03-27T06:17:15.148 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/30 file after the snapshot 2024-03-27T06:17:16.083 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/31 file after the snapshot 2024-03-27T06:17:17.132 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/32 file after the snapshot 2024-03-27T06:17:17.135 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/33 file after the snapshot 2024-03-27T06:17:17.138 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/34 file after the snapshot 2024-03-27T06:17:17.140 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/35 file after the snapshot 2024-03-27T06:17:17.142 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/36 file after the snapshot 2024-03-27T06:17:17.144 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/37 file after the snapshot 2024-03-27T06:17:17.146 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/38 file after the snapshot 2024-03-27T06:17:17.148 INFO:tasks.workunit.client.0.smithi132.stdout: create 289/39 file after the snapshot 2024-03-27T06:17:17.150 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/.snap/snap-subdir-test 2024-03-27T06:17:17.458 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/30 file after the snapshot 2024-03-27T06:17:17.459 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/31 file after the snapshot 2024-03-27T06:17:17.460 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/32 file after the snapshot 2024-03-27T06:17:17.462 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/33 file after the snapshot 2024-03-27T06:17:17.463 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/34 file after the snapshot 2024-03-27T06:17:17.464 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/35 file after the snapshot 2024-03-27T06:17:17.466 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/36 file after the snapshot 2024-03-27T06:17:17.467 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/37 file after the snapshot 2024-03-27T06:17:17.468 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/38 file after the snapshot 2024-03-27T06:17:17.470 INFO:tasks.workunit.client.0.smithi132.stdout: create 290/39 file after the snapshot 2024-03-27T06:17:17.471 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/.snap/snap-subdir-test 2024-03-27T06:17:17.520 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/30 file after the snapshot 2024-03-27T06:17:17.521 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/31 file after the snapshot 2024-03-27T06:17:17.523 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/32 file after the snapshot 2024-03-27T06:17:17.524 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/33 file after the snapshot 2024-03-27T06:17:17.525 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/34 file after the snapshot 2024-03-27T06:17:17.527 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/35 file after the snapshot 2024-03-27T06:17:17.528 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/36 file after the snapshot 2024-03-27T06:17:17.529 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/37 file after the snapshot 2024-03-27T06:17:17.530 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/38 file after the snapshot 2024-03-27T06:17:17.532 INFO:tasks.workunit.client.0.smithi132.stdout: create 291/39 file after the snapshot 2024-03-27T06:17:17.533 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/.snap/snap-subdir-test 2024-03-27T06:17:17.580 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/30 file after the snapshot 2024-03-27T06:17:17.581 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/31 file after the snapshot 2024-03-27T06:17:17.583 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/32 file after the snapshot 2024-03-27T06:17:17.584 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/33 file after the snapshot 2024-03-27T06:17:17.586 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/34 file after the snapshot 2024-03-27T06:17:17.587 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/35 file after the snapshot 2024-03-27T06:17:17.589 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/36 file after the snapshot 2024-03-27T06:17:17.590 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/37 file after the snapshot 2024-03-27T06:17:17.592 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/38 file after the snapshot 2024-03-27T06:17:17.594 INFO:tasks.workunit.client.0.smithi132.stdout: create 292/39 file after the snapshot 2024-03-27T06:17:17.596 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/.snap/snap-subdir-test 2024-03-27T06:17:17.644 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/30 file after the snapshot 2024-03-27T06:17:17.647 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/31 file after the snapshot 2024-03-27T06:17:17.649 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/32 file after the snapshot 2024-03-27T06:17:17.651 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/33 file after the snapshot 2024-03-27T06:17:17.652 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/34 file after the snapshot 2024-03-27T06:17:17.654 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/35 file after the snapshot 2024-03-27T06:17:17.656 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/36 file after the snapshot 2024-03-27T06:17:17.657 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/37 file after the snapshot 2024-03-27T06:17:17.659 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/38 file after the snapshot 2024-03-27T06:17:17.661 INFO:tasks.workunit.client.0.smithi132.stdout: create 293/39 file after the snapshot 2024-03-27T06:17:17.663 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/.snap/snap-subdir-test 2024-03-27T06:17:17.705 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/30 file after the snapshot 2024-03-27T06:17:17.706 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/31 file after the snapshot 2024-03-27T06:17:17.707 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/32 file after the snapshot 2024-03-27T06:17:17.708 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/33 file after the snapshot 2024-03-27T06:17:17.710 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/34 file after the snapshot 2024-03-27T06:17:17.711 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/35 file after the snapshot 2024-03-27T06:17:17.712 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/36 file after the snapshot 2024-03-27T06:17:17.714 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/37 file after the snapshot 2024-03-27T06:17:17.715 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/38 file after the snapshot 2024-03-27T06:17:17.716 INFO:tasks.workunit.client.0.smithi132.stdout: create 294/39 file after the snapshot 2024-03-27T06:17:17.718 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/.snap/snap-subdir-test 2024-03-27T06:17:17.762 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/30 file after the snapshot 2024-03-27T06:17:17.763 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/31 file after the snapshot 2024-03-27T06:17:17.764 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/32 file after the snapshot 2024-03-27T06:17:17.766 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/33 file after the snapshot 2024-03-27T06:17:17.767 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/34 file after the snapshot 2024-03-27T06:17:17.768 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/35 file after the snapshot 2024-03-27T06:17:17.769 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/36 file after the snapshot 2024-03-27T06:17:17.771 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/37 file after the snapshot 2024-03-27T06:17:17.772 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/38 file after the snapshot 2024-03-27T06:17:17.774 INFO:tasks.workunit.client.0.smithi132.stdout: create 295/39 file after the snapshot 2024-03-27T06:17:17.776 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/.snap/snap-subdir-test 2024-03-27T06:17:17.823 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/30 file after the snapshot 2024-03-27T06:17:17.825 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/31 file after the snapshot 2024-03-27T06:17:17.827 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/32 file after the snapshot 2024-03-27T06:17:17.828 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/33 file after the snapshot 2024-03-27T06:17:17.829 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/34 file after the snapshot 2024-03-27T06:17:17.830 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/35 file after the snapshot 2024-03-27T06:17:17.832 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/36 file after the snapshot 2024-03-27T06:17:17.834 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/37 file after the snapshot 2024-03-27T06:17:17.836 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/38 file after the snapshot 2024-03-27T06:17:17.838 INFO:tasks.workunit.client.0.smithi132.stdout: create 296/39 file after the snapshot 2024-03-27T06:17:17.840 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/.snap/snap-subdir-test 2024-03-27T06:17:17.882 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/30 file after the snapshot 2024-03-27T06:17:17.883 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/31 file after the snapshot 2024-03-27T06:17:17.885 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/32 file after the snapshot 2024-03-27T06:17:17.886 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/33 file after the snapshot 2024-03-27T06:17:17.888 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/34 file after the snapshot 2024-03-27T06:17:17.890 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/35 file after the snapshot 2024-03-27T06:17:17.891 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/36 file after the snapshot 2024-03-27T06:17:17.893 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/37 file after the snapshot 2024-03-27T06:17:17.894 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/38 file after the snapshot 2024-03-27T06:17:17.896 INFO:tasks.workunit.client.0.smithi132.stdout: create 297/39 file after the snapshot 2024-03-27T06:17:17.898 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/.snap/snap-subdir-test 2024-03-27T06:17:17.943 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/30 file after the snapshot 2024-03-27T06:17:17.944 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/31 file after the snapshot 2024-03-27T06:17:17.946 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/32 file after the snapshot 2024-03-27T06:17:17.947 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/33 file after the snapshot 2024-03-27T06:17:17.948 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/34 file after the snapshot 2024-03-27T06:17:17.949 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/35 file after the snapshot 2024-03-27T06:17:17.951 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/36 file after the snapshot 2024-03-27T06:17:17.952 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/37 file after the snapshot 2024-03-27T06:17:17.953 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/38 file after the snapshot 2024-03-27T06:17:17.955 INFO:tasks.workunit.client.0.smithi132.stdout: create 298/39 file after the snapshot 2024-03-27T06:17:17.956 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/.snap/snap-subdir-test 2024-03-27T06:17:17.991 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/30 file after the snapshot 2024-03-27T06:17:17.992 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/31 file after the snapshot 2024-03-27T06:17:17.993 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/32 file after the snapshot 2024-03-27T06:17:17.994 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/33 file after the snapshot 2024-03-27T06:17:17.996 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/34 file after the snapshot 2024-03-27T06:17:17.997 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/35 file after the snapshot 2024-03-27T06:17:17.998 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/36 file after the snapshot 2024-03-27T06:17:17.999 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/37 file after the snapshot 2024-03-27T06:17:18.000 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/38 file after the snapshot 2024-03-27T06:17:18.002 INFO:tasks.workunit.client.0.smithi132.stdout: create 299/39 file after the snapshot 2024-03-27T06:17:18.003 INFO:tasks.workunit.client.0.smithi132.stdout:Take second snapshot .snap/test2 2024-03-27T06:17:18.037 INFO:tasks.workunit.client.0.smithi132.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-03-27T06:17:18.320 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-03-27T06:17:18.321 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-03-27T06:17:18.322 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-03-27T06:17:18.323 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-03-27T06:17:18.324 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-03-27T06:17:18.325 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-03-27T06:17:18.326 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-03-27T06:17:18.557 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-03-27T06:17:18.558 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-03-27T06:17:18.559 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-03-27T06:17:18.560 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-03-27T06:17:18.561 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-03-27T06:17:18.562 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-03-27T06:17:18.563 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-03-27T06:17:18.563 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-03-27T06:17:18.563 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-03-27T06:17:18.563 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-03-27T06:17:18.822 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-03-27T06:17:18.822 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-03-27T06:17:18.822 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-03-27T06:17:18.822 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-03-27T06:17:18.823 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-03-27T06:17:18.824 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-03-27T06:17:18.825 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-03-27T06:17:18.826 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-03-27T06:17:18.827 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-03-27T06:17:18.828 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-03-27T06:17:19.059 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-03-27T06:17:19.060 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-03-27T06:17:19.061 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-03-27T06:17:19.062 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-03-27T06:17:19.063 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-03-27T06:17:19.064 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-03-27T06:17:19.065 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-03-27T06:17:19.066 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-03-27T06:17:19.066 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-03-27T06:17:19.066 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-03-27T06:17:19.066 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-03-27T06:17:19.066 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-03-27T06:17:19.290 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-03-27T06:17:19.290 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-03-27T06:17:19.290 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-03-27T06:17:19.290 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-03-27T06:17:19.291 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-03-27T06:17:19.292 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-03-27T06:17:19.293 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-03-27T06:17:19.294 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-03-27T06:17:19.295 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-03-27T06:17:19.296 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-03-27T06:17:19.516 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-03-27T06:17:19.517 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-03-27T06:17:19.518 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-03-27T06:17:19.519 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-03-27T06:17:19.520 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-03-27T06:17:19.521 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-03-27T06:17:19.522 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-03-27T06:17:19.757 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-03-27T06:17:19.758 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-03-27T06:17:19.759 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-03-27T06:17:19.760 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-03-27T06:17:19.761 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-03-27T06:17:19.762 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-03-27T06:17:19.997 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-03-27T06:17:19.998 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-03-27T06:17:19.999 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-03-27T06:17:20.000 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-03-27T06:17:20.001 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-03-27T06:17:20.002 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-03-27T06:17:20.003 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-03-27T06:17:20.003 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-03-27T06:17:20.438 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-03-27T06:17:20.439 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-03-27T06:17:20.440 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-03-27T06:17:20.441 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-03-27T06:17:20.442 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-03-27T06:17:20.443 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-03-27T06:17:20.444 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-03-27T06:17:20.685 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-03-27T06:17:20.686 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-03-27T06:17:20.687 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-03-27T06:17:20.688 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-03-27T06:17:20.689 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-03-27T06:17:20.690 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-03-27T06:17:20.691 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-03-27T06:17:20.692 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-03-27T06:17:20.692 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-03-27T06:17:20.692 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-03-27T06:17:20.692 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-03-27T06:17:20.692 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-03-27T06:17:20.942 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-03-27T06:17:20.942 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-03-27T06:17:20.942 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-03-27T06:17:20.943 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-03-27T06:17:20.944 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-03-27T06:17:20.945 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-03-27T06:17:20.946 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-03-27T06:17:20.947 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-03-27T06:17:20.948 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-03-27T06:17:21.190 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-03-27T06:17:21.191 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-03-27T06:17:21.192 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-03-27T06:17:21.193 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-03-27T06:17:21.194 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-03-27T06:17:21.195 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-03-27T06:17:21.196 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-03-27T06:17:21.413 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-03-27T06:17:21.414 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-03-27T06:17:21.415 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-03-27T06:17:21.416 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-03-27T06:17:21.417 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-03-27T06:17:21.418 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-03-27T06:17:21.658 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-03-27T06:17:21.659 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-03-27T06:17:21.660 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-03-27T06:17:21.661 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-03-27T06:17:21.662 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-03-27T06:17:21.663 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-03-27T06:17:21.664 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-03-27T06:17:21.840 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-03-27T06:17:21.840 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-03-27T06:17:21.840 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-03-27T06:17:21.841 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-03-27T06:17:21.842 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-03-27T06:17:21.843 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-03-27T06:17:21.844 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-03-27T06:17:21.845 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-03-27T06:17:21.845 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-03-27T06:17:21.845 INFO:tasks.workunit.client.0.smithi132.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-03-27T06:17:21.845 INFO:tasks.workunit.client.0.smithi132.stdout:Take third snapshot .snap/test3 2024-03-27T06:17:22.082 INFO:tasks.workunit.client.0.smithi132.stdout:Delete the snapshots... 2024-03-27T06:17:22.620 INFO:tasks.workunit.client.0.smithi132.stdout:.//.snap/test1 2024-03-27T06:17:22.620 INFO:tasks.workunit.client.0.smithi132.stdout:.//.snap/test2 2024-03-27T06:17:22.621 INFO:tasks.workunit.client.0.smithi132.stdout:.//.snap/test3 2024-03-27T06:17:22.630 INFO:tasks.workunit.client.0.smithi132.stdout:./274/.snap/snap-subdir-test 2024-03-27T06:17:22.639 INFO:tasks.workunit.client.0.smithi132.stdout:./166/.snap/snap-subdir-test 2024-03-27T06:17:22.649 INFO:tasks.workunit.client.0.smithi132.stdout:./256/.snap/snap-subdir-test 2024-03-27T06:17:22.660 INFO:tasks.workunit.client.0.smithi132.stdout:./247/.snap/snap-subdir-test 2024-03-27T06:17:22.668 INFO:tasks.workunit.client.0.smithi132.stdout:./248/.snap/snap-subdir-test 2024-03-27T06:17:22.676 INFO:tasks.workunit.client.0.smithi132.stdout:./147/.snap/snap-subdir-test 2024-03-27T06:17:22.684 INFO:tasks.workunit.client.0.smithi132.stdout:./151/.snap/snap-subdir-test 2024-03-27T06:17:22.691 INFO:tasks.workunit.client.0.smithi132.stdout:./170/.snap/snap-subdir-test 2024-03-27T06:17:22.700 INFO:tasks.workunit.client.0.smithi132.stdout:./167/.snap/snap-subdir-test 2024-03-27T06:17:22.708 INFO:tasks.workunit.client.0.smithi132.stdout:./140/.snap/snap-subdir-test 2024-03-27T06:17:22.716 INFO:tasks.workunit.client.0.smithi132.stdout:./142/.snap/snap-subdir-test 2024-03-27T06:17:22.726 INFO:tasks.workunit.client.0.smithi132.stdout:./150/.snap/snap-subdir-test 2024-03-27T06:17:22.737 INFO:tasks.workunit.client.0.smithi132.stdout:./273/.snap/snap-subdir-test 2024-03-27T06:17:22.746 INFO:tasks.workunit.client.0.smithi132.stdout:./269/.snap/snap-subdir-test 2024-03-27T06:17:22.754 INFO:tasks.workunit.client.0.smithi132.stdout:./105/.snap/snap-subdir-test 2024-03-27T06:17:22.765 INFO:tasks.workunit.client.0.smithi132.stdout:./210/.snap/snap-subdir-test 2024-03-27T06:17:22.775 INFO:tasks.workunit.client.0.smithi132.stdout:./265/.snap/snap-subdir-test 2024-03-27T06:17:22.785 INFO:tasks.workunit.client.0.smithi132.stdout:./218/.snap/snap-subdir-test 2024-03-27T06:17:22.797 INFO:tasks.workunit.client.0.smithi132.stdout:./290/.snap/snap-subdir-test 2024-03-27T06:17:22.806 INFO:tasks.workunit.client.0.smithi132.stdout:./149/.snap/snap-subdir-test 2024-03-27T06:17:22.814 INFO:tasks.workunit.client.0.smithi132.stdout:./109/.snap/snap-subdir-test 2024-03-27T06:17:22.822 INFO:tasks.workunit.client.0.smithi132.stdout:./292/.snap/snap-subdir-test 2024-03-27T06:17:22.830 INFO:tasks.workunit.client.0.smithi132.stdout:./242/.snap/snap-subdir-test 2024-03-27T06:17:22.839 INFO:tasks.workunit.client.0.smithi132.stdout:./158/.snap/snap-subdir-test 2024-03-27T06:17:22.849 INFO:tasks.workunit.client.0.smithi132.stdout:./243/.snap/snap-subdir-test 2024-03-27T06:17:22.858 INFO:tasks.workunit.client.0.smithi132.stdout:./106/.snap/snap-subdir-test 2024-03-27T06:17:22.869 INFO:tasks.workunit.client.0.smithi132.stdout:./250/.snap/snap-subdir-test 2024-03-27T06:17:22.877 INFO:tasks.workunit.client.0.smithi132.stdout:./221/.snap/snap-subdir-test 2024-03-27T06:17:22.885 INFO:tasks.workunit.client.0.smithi132.stdout:./244/.snap/snap-subdir-test 2024-03-27T06:17:22.895 INFO:tasks.workunit.client.0.smithi132.stdout:./258/.snap/snap-subdir-test 2024-03-27T06:17:22.904 INFO:tasks.workunit.client.0.smithi132.stdout:./268/.snap/snap-subdir-test 2024-03-27T06:17:22.912 INFO:tasks.workunit.client.0.smithi132.stdout:./189/.snap/snap-subdir-test 2024-03-27T06:17:22.922 INFO:tasks.workunit.client.0.smithi132.stdout:./285/.snap/snap-subdir-test 2024-03-27T06:17:22.933 INFO:tasks.workunit.client.0.smithi132.stdout:./208/.snap/snap-subdir-test 2024-03-27T06:17:22.944 INFO:tasks.workunit.client.0.smithi132.stdout:./288/.snap/snap-subdir-test 2024-03-27T06:17:22.955 INFO:tasks.workunit.client.0.smithi132.stdout:./209/.snap/snap-subdir-test 2024-03-27T06:17:22.966 INFO:tasks.workunit.client.0.smithi132.stdout:./253/.snap/snap-subdir-test 2024-03-27T06:17:22.979 INFO:tasks.workunit.client.0.smithi132.stdout:./249/.snap/snap-subdir-test 2024-03-27T06:17:22.989 INFO:tasks.workunit.client.0.smithi132.stdout:./133/.snap/snap-subdir-test 2024-03-27T06:17:23.001 INFO:tasks.workunit.client.0.smithi132.stdout:./291/.snap/snap-subdir-test 2024-03-27T06:17:23.012 INFO:tasks.workunit.client.0.smithi132.stdout:./226/.snap/snap-subdir-test 2024-03-27T06:17:23.021 INFO:tasks.workunit.client.0.smithi132.stdout:./173/.snap/snap-subdir-test 2024-03-27T06:17:23.031 INFO:tasks.workunit.client.0.smithi132.stdout:./174/.snap/snap-subdir-test 2024-03-27T06:17:23.044 INFO:tasks.workunit.client.0.smithi132.stdout:./299/.snap/snap-subdir-test 2024-03-27T06:17:23.056 INFO:tasks.workunit.client.0.smithi132.stdout:./262/.snap/snap-subdir-test 2024-03-27T06:17:23.067 INFO:tasks.workunit.client.0.smithi132.stdout:./231/.snap/snap-subdir-test 2024-03-27T06:17:23.079 INFO:tasks.workunit.client.0.smithi132.stdout:./214/.snap/snap-subdir-test 2024-03-27T06:17:23.090 INFO:tasks.workunit.client.0.smithi132.stdout:./281/.snap/snap-subdir-test 2024-03-27T06:17:23.101 INFO:tasks.workunit.client.0.smithi132.stdout:./289/.snap/snap-subdir-test 2024-03-27T06:17:23.111 INFO:tasks.workunit.client.0.smithi132.stdout:./283/.snap/snap-subdir-test 2024-03-27T06:17:23.123 INFO:tasks.workunit.client.0.smithi132.stdout:./232/.snap/snap-subdir-test 2024-03-27T06:17:23.133 INFO:tasks.workunit.client.0.smithi132.stdout:./298/.snap/snap-subdir-test 2024-03-27T06:17:23.142 INFO:tasks.workunit.client.0.smithi132.stdout:./233/.snap/snap-subdir-test 2024-03-27T06:17:23.151 INFO:tasks.workunit.client.0.smithi132.stdout:./155/.snap/snap-subdir-test 2024-03-27T06:17:23.160 INFO:tasks.workunit.client.0.smithi132.stdout:./114/.snap/snap-subdir-test 2024-03-27T06:17:23.170 INFO:tasks.workunit.client.0.smithi132.stdout:./159/.snap/snap-subdir-test 2024-03-27T06:17:23.180 INFO:tasks.workunit.client.0.smithi132.stdout:./187/.snap/snap-subdir-test 2024-03-27T06:17:23.190 INFO:tasks.workunit.client.0.smithi132.stdout:./185/.snap/snap-subdir-test 2024-03-27T06:17:23.203 INFO:tasks.workunit.client.0.smithi132.stdout:./293/.snap/snap-subdir-test 2024-03-27T06:17:23.214 INFO:tasks.workunit.client.0.smithi132.stdout:./228/.snap/snap-subdir-test 2024-03-27T06:17:23.223 INFO:tasks.workunit.client.0.smithi132.stdout:./120/.snap/snap-subdir-test 2024-03-27T06:17:23.232 INFO:tasks.workunit.client.0.smithi132.stdout:./110/.snap/snap-subdir-test 2024-03-27T06:17:23.242 INFO:tasks.workunit.client.0.smithi132.stdout:./267/.snap/snap-subdir-test 2024-03-27T06:17:23.254 INFO:tasks.workunit.client.0.smithi132.stdout:./202/.snap/snap-subdir-test 2024-03-27T06:17:23.266 INFO:tasks.workunit.client.0.smithi132.stdout:./137/.snap/snap-subdir-test 2024-03-27T06:17:23.276 INFO:tasks.workunit.client.0.smithi132.stdout:./246/.snap/snap-subdir-test 2024-03-27T06:17:23.288 INFO:tasks.workunit.client.0.smithi132.stdout:./219/.snap/snap-subdir-test 2024-03-27T06:17:23.297 INFO:tasks.workunit.client.0.smithi132.stdout:./198/.snap/snap-subdir-test 2024-03-27T06:17:23.308 INFO:tasks.workunit.client.0.smithi132.stdout:./136/.snap/snap-subdir-test 2024-03-27T06:17:23.321 INFO:tasks.workunit.client.0.smithi132.stdout:./270/.snap/snap-subdir-test 2024-03-27T06:17:23.333 INFO:tasks.workunit.client.0.smithi132.stdout:./125/.snap/snap-subdir-test 2024-03-27T06:17:23.345 INFO:tasks.workunit.client.0.smithi132.stdout:./194/.snap/snap-subdir-test 2024-03-27T06:17:23.355 INFO:tasks.workunit.client.0.smithi132.stdout:./119/.snap/snap-subdir-test 2024-03-27T06:17:23.368 INFO:tasks.workunit.client.0.smithi132.stdout:./297/.snap/snap-subdir-test 2024-03-27T06:17:23.380 INFO:tasks.workunit.client.0.smithi132.stdout:./230/.snap/snap-subdir-test 2024-03-27T06:17:23.391 INFO:tasks.workunit.client.0.smithi132.stdout:./260/.snap/snap-subdir-test 2024-03-27T06:17:23.400 INFO:tasks.workunit.client.0.smithi132.stdout:./157/.snap/snap-subdir-test 2024-03-27T06:17:23.409 INFO:tasks.workunit.client.0.smithi132.stdout:./172/.snap/snap-subdir-test 2024-03-27T06:17:23.421 INFO:tasks.workunit.client.0.smithi132.stdout:./277/.snap/snap-subdir-test 2024-03-27T06:17:23.430 INFO:tasks.workunit.client.0.smithi132.stdout:./186/.snap/snap-subdir-test 2024-03-27T06:17:23.440 INFO:tasks.workunit.client.0.smithi132.stdout:./237/.snap/snap-subdir-test 2024-03-27T06:17:23.449 INFO:tasks.workunit.client.0.smithi132.stdout:./190/.snap/snap-subdir-test 2024-03-27T06:17:23.460 INFO:tasks.workunit.client.0.smithi132.stdout:./213/.snap/snap-subdir-test 2024-03-27T06:17:23.468 INFO:tasks.workunit.client.0.smithi132.stdout:./104/.snap/snap-subdir-test 2024-03-27T06:17:23.479 INFO:tasks.workunit.client.0.smithi132.stdout:./164/.snap/snap-subdir-test 2024-03-27T06:17:23.487 INFO:tasks.workunit.client.0.smithi132.stdout:./112/.snap/snap-subdir-test 2024-03-27T06:17:23.498 INFO:tasks.workunit.client.0.smithi132.stdout:./295/.snap/snap-subdir-test 2024-03-27T06:17:23.510 INFO:tasks.workunit.client.0.smithi132.stdout:./272/.snap/snap-subdir-test 2024-03-27T06:17:23.521 INFO:tasks.workunit.client.0.smithi132.stdout:./100/.snap/snap-subdir-test 2024-03-27T06:17:23.530 INFO:tasks.workunit.client.0.smithi132.stdout:./134/.snap/snap-subdir-test 2024-03-27T06:17:23.541 INFO:tasks.workunit.client.0.smithi132.stdout:./200/.snap/snap-subdir-test 2024-03-27T06:17:23.553 INFO:tasks.workunit.client.0.smithi132.stdout:./115/.snap/snap-subdir-test 2024-03-27T06:17:23.566 INFO:tasks.workunit.client.0.smithi132.stdout:./284/.snap/snap-subdir-test 2024-03-27T06:17:23.580 INFO:tasks.workunit.client.0.smithi132.stdout:./280/.snap/snap-subdir-test 2024-03-27T06:17:23.592 INFO:tasks.workunit.client.0.smithi132.stdout:./116/.snap/snap-subdir-test 2024-03-27T06:17:23.604 INFO:tasks.workunit.client.0.smithi132.stdout:./245/.snap/snap-subdir-test 2024-03-27T06:17:23.613 INFO:tasks.workunit.client.0.smithi132.stdout:./195/.snap/snap-subdir-test 2024-03-27T06:17:23.622 INFO:tasks.workunit.client.0.smithi132.stdout:./138/.snap/snap-subdir-test 2024-03-27T06:17:23.632 INFO:tasks.workunit.client.0.smithi132.stdout:./135/.snap/snap-subdir-test 2024-03-27T06:17:23.641 INFO:tasks.workunit.client.0.smithi132.stdout:./117/.snap/snap-subdir-test 2024-03-27T06:17:23.650 INFO:tasks.workunit.client.0.smithi132.stdout:./241/.snap/snap-subdir-test 2024-03-27T06:17:23.658 INFO:tasks.workunit.client.0.smithi132.stdout:./143/.snap/snap-subdir-test 2024-03-27T06:17:23.665 INFO:tasks.workunit.client.0.smithi132.stdout:./180/.snap/snap-subdir-test 2024-03-27T06:17:23.674 INFO:tasks.workunit.client.0.smithi132.stdout:./126/.snap/snap-subdir-test 2024-03-27T06:17:23.682 INFO:tasks.workunit.client.0.smithi132.stdout:./161/.snap/snap-subdir-test 2024-03-27T06:17:23.691 INFO:tasks.workunit.client.0.smithi132.stdout:./204/.snap/snap-subdir-test 2024-03-27T06:17:23.700 INFO:tasks.workunit.client.0.smithi132.stdout:./176/.snap/snap-subdir-test 2024-03-27T06:17:23.710 INFO:tasks.workunit.client.0.smithi132.stdout:./255/.snap/snap-subdir-test 2024-03-27T06:17:23.720 INFO:tasks.workunit.client.0.smithi132.stdout:./215/.snap/snap-subdir-test 2024-03-27T06:17:23.729 INFO:tasks.workunit.client.0.smithi132.stdout:./111/.snap/snap-subdir-test 2024-03-27T06:17:23.738 INFO:tasks.workunit.client.0.smithi132.stdout:./238/.snap/snap-subdir-test 2024-03-27T06:17:23.746 INFO:tasks.workunit.client.0.smithi132.stdout:./191/.snap/snap-subdir-test 2024-03-27T06:17:23.755 INFO:tasks.workunit.client.0.smithi132.stdout:./184/.snap/snap-subdir-test 2024-03-27T06:17:23.762 INFO:tasks.workunit.client.0.smithi132.stdout:./192/.snap/snap-subdir-test 2024-03-27T06:17:23.770 INFO:tasks.workunit.client.0.smithi132.stdout:./103/.snap/snap-subdir-test 2024-03-27T06:17:23.779 INFO:tasks.workunit.client.0.smithi132.stdout:./128/.snap/snap-subdir-test 2024-03-27T06:17:23.791 INFO:tasks.workunit.client.0.smithi132.stdout:./229/.snap/snap-subdir-test 2024-03-27T06:17:23.800 INFO:tasks.workunit.client.0.smithi132.stdout:./144/.snap/snap-subdir-test 2024-03-27T06:17:23.810 INFO:tasks.workunit.client.0.smithi132.stdout:./146/.snap/snap-subdir-test 2024-03-27T06:17:23.819 INFO:tasks.workunit.client.0.smithi132.stdout:./227/.snap/snap-subdir-test 2024-03-27T06:17:23.828 INFO:tasks.workunit.client.0.smithi132.stdout:./278/.snap/snap-subdir-test 2024-03-27T06:17:23.837 INFO:tasks.workunit.client.0.smithi132.stdout:./141/.snap/snap-subdir-test 2024-03-27T06:17:23.847 INFO:tasks.workunit.client.0.smithi132.stdout:./234/.snap/snap-subdir-test 2024-03-27T06:17:23.855 INFO:tasks.workunit.client.0.smithi132.stdout:./225/.snap/snap-subdir-test 2024-03-27T06:17:23.865 INFO:tasks.workunit.client.0.smithi132.stdout:./182/.snap/snap-subdir-test 2024-03-27T06:17:23.872 INFO:tasks.workunit.client.0.smithi132.stdout:./107/.snap/snap-subdir-test 2024-03-27T06:17:23.880 INFO:tasks.workunit.client.0.smithi132.stdout:./178/.snap/snap-subdir-test 2024-03-27T06:17:23.890 INFO:tasks.workunit.client.0.smithi132.stdout:./266/.snap/snap-subdir-test 2024-03-27T06:17:23.898 INFO:tasks.workunit.client.0.smithi132.stdout:./240/.snap/snap-subdir-test 2024-03-27T06:17:23.907 INFO:tasks.workunit.client.0.smithi132.stdout:./156/.snap/snap-subdir-test 2024-03-27T06:17:23.915 INFO:tasks.workunit.client.0.smithi132.stdout:./131/.snap/snap-subdir-test 2024-03-27T06:17:23.922 INFO:tasks.workunit.client.0.smithi132.stdout:./123/.snap/snap-subdir-test 2024-03-27T06:17:23.930 INFO:tasks.workunit.client.0.smithi132.stdout:./160/.snap/snap-subdir-test 2024-03-27T06:17:23.939 INFO:tasks.workunit.client.0.smithi132.stdout:./165/.snap/snap-subdir-test 2024-03-27T06:17:23.946 INFO:tasks.workunit.client.0.smithi132.stdout:./101/.snap/snap-subdir-test 2024-03-27T06:17:23.955 INFO:tasks.workunit.client.0.smithi132.stdout:./148/.snap/snap-subdir-test 2024-03-27T06:17:23.962 INFO:tasks.workunit.client.0.smithi132.stdout:./168/.snap/snap-subdir-test 2024-03-27T06:17:23.970 INFO:tasks.workunit.client.0.smithi132.stdout:./179/.snap/snap-subdir-test 2024-03-27T06:17:23.978 INFO:tasks.workunit.client.0.smithi132.stdout:./257/.snap/snap-subdir-test 2024-03-27T06:17:23.986 INFO:tasks.workunit.client.0.smithi132.stdout:./252/.snap/snap-subdir-test 2024-03-27T06:17:23.994 INFO:tasks.workunit.client.0.smithi132.stdout:./132/.snap/snap-subdir-test 2024-03-27T06:17:24.003 INFO:tasks.workunit.client.0.smithi132.stdout:./236/.snap/snap-subdir-test 2024-03-27T06:17:24.012 INFO:tasks.workunit.client.0.smithi132.stdout:./211/.snap/snap-subdir-test 2024-03-27T06:17:24.020 INFO:tasks.workunit.client.0.smithi132.stdout:./199/.snap/snap-subdir-test 2024-03-27T06:17:24.029 INFO:tasks.workunit.client.0.smithi132.stdout:./205/.snap/snap-subdir-test 2024-03-27T06:17:24.037 INFO:tasks.workunit.client.0.smithi132.stdout:./163/.snap/snap-subdir-test 2024-03-27T06:17:24.046 INFO:tasks.workunit.client.0.smithi132.stdout:./261/.snap/snap-subdir-test 2024-03-27T06:17:24.055 INFO:tasks.workunit.client.0.smithi132.stdout:./201/.snap/snap-subdir-test 2024-03-27T06:17:24.064 INFO:tasks.workunit.client.0.smithi132.stdout:./282/.snap/snap-subdir-test 2024-03-27T06:17:24.072 INFO:tasks.workunit.client.0.smithi132.stdout:./296/.snap/snap-subdir-test 2024-03-27T06:17:24.080 INFO:tasks.workunit.client.0.smithi132.stdout:./217/.snap/snap-subdir-test 2024-03-27T06:17:24.088 INFO:tasks.workunit.client.0.smithi132.stdout:./153/.snap/snap-subdir-test 2024-03-27T06:17:24.097 INFO:tasks.workunit.client.0.smithi132.stdout:./206/.snap/snap-subdir-test 2024-03-27T06:17:24.106 INFO:tasks.workunit.client.0.smithi132.stdout:./122/.snap/snap-subdir-test 2024-03-27T06:17:24.115 INFO:tasks.workunit.client.0.smithi132.stdout:./162/.snap/snap-subdir-test 2024-03-27T06:17:24.122 INFO:tasks.workunit.client.0.smithi132.stdout:./197/.snap/snap-subdir-test 2024-03-27T06:17:24.133 INFO:tasks.workunit.client.0.smithi132.stdout:./216/.snap/snap-subdir-test 2024-03-27T06:17:24.792 INFO:tasks.workunit.client.0.smithi132.stdout:./254/.snap/snap-subdir-test 2024-03-27T06:17:24.802 INFO:tasks.workunit.client.0.smithi132.stdout:./113/.snap/snap-subdir-test 2024-03-27T06:17:24.810 INFO:tasks.workunit.client.0.smithi132.stdout:./152/.snap/snap-subdir-test 2024-03-27T06:17:24.818 INFO:tasks.workunit.client.0.smithi132.stdout:./121/.snap/snap-subdir-test 2024-03-27T06:17:24.827 INFO:tasks.workunit.client.0.smithi132.stdout:./118/.snap/snap-subdir-test 2024-03-27T06:17:24.836 INFO:tasks.workunit.client.0.smithi132.stdout:./196/.snap/snap-subdir-test 2024-03-27T06:17:24.844 INFO:tasks.workunit.client.0.smithi132.stdout:./124/.snap/snap-subdir-test 2024-03-27T06:17:24.854 INFO:tasks.workunit.client.0.smithi132.stdout:./239/.snap/snap-subdir-test 2024-03-27T06:17:24.862 INFO:tasks.workunit.client.0.smithi132.stdout:./188/.snap/snap-subdir-test 2024-03-27T06:17:24.871 INFO:tasks.workunit.client.0.smithi132.stdout:./287/.snap/snap-subdir-test 2024-03-27T06:17:24.881 INFO:tasks.workunit.client.0.smithi132.stdout:./275/.snap/snap-subdir-test 2024-03-27T06:17:24.891 INFO:tasks.workunit.client.0.smithi132.stdout:./259/.snap/snap-subdir-test 2024-03-27T06:17:24.901 INFO:tasks.workunit.client.0.smithi132.stdout:./224/.snap/snap-subdir-test 2024-03-27T06:17:24.910 INFO:tasks.workunit.client.0.smithi132.stdout:./264/.snap/snap-subdir-test 2024-03-27T06:17:24.919 INFO:tasks.workunit.client.0.smithi132.stdout:./145/.snap/snap-subdir-test 2024-03-27T06:17:24.928 INFO:tasks.workunit.client.0.smithi132.stdout:./279/.snap/snap-subdir-test 2024-03-27T06:17:24.940 INFO:tasks.workunit.client.0.smithi132.stdout:./263/.snap/snap-subdir-test 2024-03-27T06:17:24.949 INFO:tasks.workunit.client.0.smithi132.stdout:./130/.snap/snap-subdir-test 2024-03-27T06:17:24.960 INFO:tasks.workunit.client.0.smithi132.stdout:./193/.snap/snap-subdir-test 2024-03-27T06:17:24.972 INFO:tasks.workunit.client.0.smithi132.stdout:./223/.snap/snap-subdir-test 2024-03-27T06:17:24.984 INFO:tasks.workunit.client.0.smithi132.stdout:./207/.snap/snap-subdir-test 2024-03-27T06:17:24.996 INFO:tasks.workunit.client.0.smithi132.stdout:./276/.snap/snap-subdir-test 2024-03-27T06:17:25.008 INFO:tasks.workunit.client.0.smithi132.stdout:./271/.snap/snap-subdir-test 2024-03-27T06:17:25.019 INFO:tasks.workunit.client.0.smithi132.stdout:./251/.snap/snap-subdir-test 2024-03-27T06:17:25.029 INFO:tasks.workunit.client.0.smithi132.stdout:./175/.snap/snap-subdir-test 2024-03-27T06:17:25.039 INFO:tasks.workunit.client.0.smithi132.stdout:./222/.snap/snap-subdir-test 2024-03-27T06:17:25.047 INFO:tasks.workunit.client.0.smithi132.stdout:./220/.snap/snap-subdir-test 2024-03-27T06:17:25.054 INFO:tasks.workunit.client.0.smithi132.stdout:./203/.snap/snap-subdir-test 2024-03-27T06:17:25.063 INFO:tasks.workunit.client.0.smithi132.stdout:./181/.snap/snap-subdir-test 2024-03-27T06:17:25.074 INFO:tasks.workunit.client.0.smithi132.stdout:./286/.snap/snap-subdir-test 2024-03-27T06:17:25.083 INFO:tasks.workunit.client.0.smithi132.stdout:./127/.snap/snap-subdir-test 2024-03-27T06:17:25.091 INFO:tasks.workunit.client.0.smithi132.stdout:./108/.snap/snap-subdir-test 2024-03-27T06:17:25.100 INFO:tasks.workunit.client.0.smithi132.stdout:./183/.snap/snap-subdir-test 2024-03-27T06:17:25.109 INFO:tasks.workunit.client.0.smithi132.stdout:./294/.snap/snap-subdir-test 2024-03-27T06:17:25.120 INFO:tasks.workunit.client.0.smithi132.stdout:./235/.snap/snap-subdir-test 2024-03-27T06:17:25.131 INFO:tasks.workunit.client.0.smithi132.stdout:./177/.snap/snap-subdir-test 2024-03-27T06:17:25.141 INFO:tasks.workunit.client.0.smithi132.stdout:./169/.snap/snap-subdir-test 2024-03-27T06:17:25.152 INFO:tasks.workunit.client.0.smithi132.stdout:./212/.snap/snap-subdir-test 2024-03-27T06:17:25.160 INFO:tasks.workunit.client.0.smithi132.stdout:./154/.snap/snap-subdir-test 2024-03-27T06:17:25.169 INFO:tasks.workunit.client.0.smithi132.stdout:./171/.snap/snap-subdir-test 2024-03-27T06:17:25.179 INFO:tasks.workunit.client.0.smithi132.stdout:./139/.snap/snap-subdir-test 2024-03-27T06:17:25.188 INFO:tasks.workunit.client.0.smithi132.stdout:./102/.snap/snap-subdir-test 2024-03-27T06:17:25.198 INFO:tasks.workunit.client.0.smithi132.stdout:./129/.snap/snap-subdir-test 2024-03-27T06:17:27.074 INFO:tasks.workunit.client.0.smithi132.stdout:Delete all the files and directories ... 2024-03-27T06:17:27.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/19' 2024-03-27T06:17:27.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/15' 2024-03-27T06:17:27.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/31' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/36' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/20' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/12' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/39' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/10' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/34' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/38' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/11' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/16' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/32' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/30' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/14' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/18' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/13' 2024-03-27T06:17:27.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/33' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/17' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/35' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './100/37' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './100' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/19' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/15' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/31' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/36' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/20' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/12' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/39' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/10' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/34' 2024-03-27T06:17:27.686 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/38' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/11' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/16' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/32' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/30' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/14' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/18' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/13' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/33' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/17' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/35' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './101/37' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './101' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/19' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/15' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/31' 2024-03-27T06:17:27.687 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/36' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/20' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/12' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/39' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/10' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/34' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/38' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/11' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/16' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/32' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/30' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/14' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/18' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/13' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/33' 2024-03-27T06:17:27.688 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/17' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/35' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './102/37' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './102' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/19' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/15' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/31' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/36' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/20' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/12' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/39' 2024-03-27T06:17:27.689 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/10' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/34' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/38' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/11' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/16' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/32' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/30' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/14' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/18' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/13' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/33' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/17' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/35' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed './103/37' 2024-03-27T06:17:27.690 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './103' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/19' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/15' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/31' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/36' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/20' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/12' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/39' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/10' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/34' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/38' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/11' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/16' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/32' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/30' 2024-03-27T06:17:27.691 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/14' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/18' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/13' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/33' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/17' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/35' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './104/37' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './104' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/19' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/15' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/31' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/36' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/20' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/12' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/39' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/10' 2024-03-27T06:17:27.692 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/34' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/38' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/11' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/16' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/32' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/30' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/14' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/18' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/13' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/33' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/17' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/35' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './105/37' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './105' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/19' 2024-03-27T06:17:27.693 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/15' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/31' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/36' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/20' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/12' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/39' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/10' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/34' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/38' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/11' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/16' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/32' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/30' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/14' 2024-03-27T06:17:27.694 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/18' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/13' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/33' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/17' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/35' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './106/37' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './106' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/19' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/15' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/31' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/36' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/20' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/12' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/39' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/10' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/34' 2024-03-27T06:17:27.695 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/38' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/11' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/16' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/32' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/30' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/14' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/18' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/13' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/33' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/17' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/35' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './107/37' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './107' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/19' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/15' 2024-03-27T06:17:27.696 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/31' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/36' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/20' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/12' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/39' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/10' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/34' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/38' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/11' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/16' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/32' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/30' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/14' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/18' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/13' 2024-03-27T06:17:27.697 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/33' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/17' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/35' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './108/37' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './108' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/19' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/15' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/31' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/36' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/20' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/12' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/39' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/10' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/34' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/38' 2024-03-27T06:17:27.698 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/11' 2024-03-27T06:17:27.699 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/16' 2024-03-27T06:17:27.699 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/32' 2024-03-27T06:17:27.699 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/30' 2024-03-27T06:17:28.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/14' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/18' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/13' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/33' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/17' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/35' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './109/37' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './109' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/19' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/15' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/31' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/36' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/20' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/12' 2024-03-27T06:17:28.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/39' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/10' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/34' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/38' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/11' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/16' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/32' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/30' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/14' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/18' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/13' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/33' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/17' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/35' 2024-03-27T06:17:28.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './110/37' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './110' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/19' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/15' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/31' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/36' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/20' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/12' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/39' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/10' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/34' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/38' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/11' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/16' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/32' 2024-03-27T06:17:28.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/30' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/14' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/18' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/13' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/33' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/17' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/35' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './111/37' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './111' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/19' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/15' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/31' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/36' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/20' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/12' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/39' 2024-03-27T06:17:28.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/10' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/34' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/38' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/11' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/16' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/32' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/30' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/14' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/18' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/13' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/33' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/17' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/35' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './112/37' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './112' 2024-03-27T06:17:28.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/19' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/15' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/31' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/36' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/20' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/12' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/39' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/10' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/34' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/38' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/11' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/16' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/32' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/30' 2024-03-27T06:17:28.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/14' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/18' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/13' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/33' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/17' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/35' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './113/37' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './113' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/19' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/15' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/31' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/36' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/20' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/12' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/39' 2024-03-27T06:17:28.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/10' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/34' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/38' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/11' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/16' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/32' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/30' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/14' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/18' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/13' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/33' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/17' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/35' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './114/37' 2024-03-27T06:17:28.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './114' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/19' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/15' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/31' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/36' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/20' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/12' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/39' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/10' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/34' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/38' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/11' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/16' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/32' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/30' 2024-03-27T06:17:28.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/14' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/18' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/13' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/33' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/17' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/35' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './115/37' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './115' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/19' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/15' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/31' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/36' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/20' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/12' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/39' 2024-03-27T06:17:28.322 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/10' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/34' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/38' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/11' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/16' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/32' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/30' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/14' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/18' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/13' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/33' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/17' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/35' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './116/37' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './116' 2024-03-27T06:17:28.323 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/19' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/15' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/31' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/36' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/20' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/12' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/39' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/10' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/34' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/38' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/11' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/16' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/32' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/30' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/14' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/18' 2024-03-27T06:17:28.324 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/13' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/33' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/17' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/35' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './117/37' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './117' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/19' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/15' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/31' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/36' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/20' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/12' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/39' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/10' 2024-03-27T06:17:28.325 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/34' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/38' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/11' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/16' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/32' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/30' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/14' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/18' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/13' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/33' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/17' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/35' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './118/37' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './118' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/19' 2024-03-27T06:17:28.326 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/15' 2024-03-27T06:17:28.327 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/31' 2024-03-27T06:17:28.327 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/36' 2024-03-27T06:17:28.327 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/20' 2024-03-27T06:17:28.327 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/12' 2024-03-27T06:17:29.390 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/39' 2024-03-27T06:17:29.390 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/10' 2024-03-27T06:17:29.390 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/34' 2024-03-27T06:17:29.390 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/38' 2024-03-27T06:17:29.390 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/11' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/16' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/32' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/30' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/14' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/18' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/13' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/33' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/17' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/35' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './119/37' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './119' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/19' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/15' 2024-03-27T06:17:29.391 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/31' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/36' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/20' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/12' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/39' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/10' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/34' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/38' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/11' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/16' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/32' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/30' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/14' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/18' 2024-03-27T06:17:29.392 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/13' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/33' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/17' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/35' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './120/37' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './120' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/19' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/15' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/31' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/36' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/20' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/12' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/39' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/10' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/34' 2024-03-27T06:17:29.393 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/38' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/11' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/16' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/32' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/30' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/14' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/18' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/13' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/33' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/17' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/35' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './121/37' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './121' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/19' 2024-03-27T06:17:29.394 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/15' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/31' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/36' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/20' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/12' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/39' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/10' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/34' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/38' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/11' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/16' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/32' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/30' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/14' 2024-03-27T06:17:29.395 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/18' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/13' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/33' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/17' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/35' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './122/37' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './122' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/19' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/15' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/31' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/36' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/20' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/12' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/39' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/10' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/34' 2024-03-27T06:17:29.396 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/38' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/11' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/16' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/32' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/30' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/14' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/18' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/13' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/33' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/17' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/35' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './123/37' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './123' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/19' 2024-03-27T06:17:29.397 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/15' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/31' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/36' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/20' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/12' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/39' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/10' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/34' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/38' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/11' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/16' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/32' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/30' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/14' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/18' 2024-03-27T06:17:29.398 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/13' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/33' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/17' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/35' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './124/37' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './124' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/19' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/15' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/31' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/36' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/20' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/12' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/39' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/10' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/34' 2024-03-27T06:17:29.399 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/38' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/11' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/16' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/32' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/30' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/14' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/18' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/13' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/33' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/17' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/35' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './125/37' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './125' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/19' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/15' 2024-03-27T06:17:29.400 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/31' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/36' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/20' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/12' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/39' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/10' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/34' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/38' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/11' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/16' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/32' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/30' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/14' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/18' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/13' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/33' 2024-03-27T06:17:29.401 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/17' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/35' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './126/37' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './126' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/19' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/15' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/31' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/36' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/20' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/12' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/39' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/10' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/34' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/38' 2024-03-27T06:17:29.402 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/11' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/16' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/32' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/30' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/14' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/18' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/13' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/33' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/17' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/35' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './127/37' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './127' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/19' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/15' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/31' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/36' 2024-03-27T06:17:29.403 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/20' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/12' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/39' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/10' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/34' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/38' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/11' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/16' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/32' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/30' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/14' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/18' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/13' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/33' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/17' 2024-03-27T06:17:29.404 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/35' 2024-03-27T06:17:30.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './128/37' 2024-03-27T06:17:30.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './128' 2024-03-27T06:17:30.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/19' 2024-03-27T06:17:30.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/15' 2024-03-27T06:17:30.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/31' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/36' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/20' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/12' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/39' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/10' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/34' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/38' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/11' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/16' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/32' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/30' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/14' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/18' 2024-03-27T06:17:30.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/13' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/33' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/17' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/35' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './129/37' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './129' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/19' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/15' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/31' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/36' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/20' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/12' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/39' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/10' 2024-03-27T06:17:30.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/34' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/38' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/11' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/16' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/32' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/30' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/14' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/18' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/13' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/33' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/17' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/35' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './130/37' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './130' 2024-03-27T06:17:30.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/19' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/15' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/31' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/36' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/20' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/12' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/39' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/10' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/34' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/38' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/11' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/16' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/32' 2024-03-27T06:17:30.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/30' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/14' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/18' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/13' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/33' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/17' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/35' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './131/37' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './131' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/19' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/15' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/31' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/36' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/20' 2024-03-27T06:17:30.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/12' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/39' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/10' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/34' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/38' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/11' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/16' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/32' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/30' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/14' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/18' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/13' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/33' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/17' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/35' 2024-03-27T06:17:30.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './132/37' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './132' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/19' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/15' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/31' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/36' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/20' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/12' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/39' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/10' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/34' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/38' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/11' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/16' 2024-03-27T06:17:30.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/32' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/30' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/14' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/18' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/13' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/33' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/17' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/35' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './133/37' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './133' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/19' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/15' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/31' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/36' 2024-03-27T06:17:30.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/20' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/12' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/39' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/10' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/34' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/38' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/11' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/16' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/32' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/30' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/14' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/18' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/13' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/33' 2024-03-27T06:17:30.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/17' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/35' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './134/37' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './134' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/19' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/15' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/31' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/36' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/20' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/12' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/39' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/10' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/34' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/38' 2024-03-27T06:17:30.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/11' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/16' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/32' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/30' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/14' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/18' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/13' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/33' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/17' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/35' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './135/37' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './135' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/19' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/15' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/31' 2024-03-27T06:17:30.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/36' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/20' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/12' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/39' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/10' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/34' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/38' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/11' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/16' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/32' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/30' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/14' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/18' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/13' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/33' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/17' 2024-03-27T06:17:30.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/35' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './136/37' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './136' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/19' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/15' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/31' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/36' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/20' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/12' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/39' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/10' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/34' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/38' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/11' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/16' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/32' 2024-03-27T06:17:30.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/30' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/14' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/18' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/13' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/33' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/17' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/35' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './137/37' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './137' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/19' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/15' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/31' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/36' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/20' 2024-03-27T06:17:30.470 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/12' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/39' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/10' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/34' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/38' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/11' 2024-03-27T06:17:30.471 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/16' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/32' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/30' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/14' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/18' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/13' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/33' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/17' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/35' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed './138/37' 2024-03-27T06:17:31.204 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './138' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/19' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/15' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/31' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/36' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/20' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/12' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/39' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/10' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/34' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/38' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/11' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/16' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/32' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/30' 2024-03-27T06:17:31.205 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/14' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/18' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/13' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/33' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/17' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/35' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './139/37' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './139' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/19' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/15' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/31' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/36' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/20' 2024-03-27T06:17:31.206 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/12' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/39' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/10' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/34' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/38' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/11' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/16' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/32' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/30' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/14' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/18' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/13' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/33' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/17' 2024-03-27T06:17:31.207 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/35' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './140/37' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './140' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/19' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/15' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/31' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/36' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/20' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/12' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/39' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/10' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/34' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/38' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/11' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/16' 2024-03-27T06:17:31.208 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/32' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/30' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/14' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/18' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/13' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/33' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/17' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/35' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './141/37' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './141' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/19' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/15' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/31' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/36' 2024-03-27T06:17:31.209 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/20' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/12' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/39' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/10' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/34' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/38' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/11' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/16' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/32' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/30' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/14' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/18' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/13' 2024-03-27T06:17:31.210 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/33' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/17' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/35' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './142/37' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './142' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/19' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/15' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/31' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/36' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/20' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/12' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/39' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/10' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/34' 2024-03-27T06:17:31.211 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/38' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/11' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/16' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/32' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/30' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/14' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/18' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/13' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/33' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/17' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/35' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './143/37' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './143' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/19' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/15' 2024-03-27T06:17:31.212 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/31' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/36' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/20' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/12' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/39' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/10' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/34' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/38' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/11' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/16' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/32' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/30' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/14' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/18' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/13' 2024-03-27T06:17:31.213 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/33' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/17' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/35' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './144/37' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './144' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/19' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/15' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/31' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/36' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/20' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/12' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/39' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/10' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/34' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/38' 2024-03-27T06:17:31.214 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/11' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/16' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/32' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/30' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/14' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/18' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/13' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/33' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/17' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/35' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './145/37' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './145' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/19' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/15' 2024-03-27T06:17:31.215 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/31' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/36' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/20' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/12' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/39' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/10' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/34' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/38' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/11' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/16' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/32' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/30' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/14' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/18' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/13' 2024-03-27T06:17:31.216 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/33' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/17' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/35' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './146/37' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './146' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/19' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/15' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/31' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/36' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/20' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/12' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/39' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/10' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/34' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/38' 2024-03-27T06:17:31.217 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/11' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/16' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/32' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/30' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/14' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/18' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/13' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/33' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/17' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/35' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './147/37' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './147' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/19' 2024-03-27T06:17:31.218 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/15' 2024-03-27T06:17:31.219 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/31' 2024-03-27T06:17:31.219 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/36' 2024-03-27T06:17:32.417 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/20' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/12' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/39' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/10' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/34' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/38' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/11' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/16' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/32' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/30' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/14' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/18' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/13' 2024-03-27T06:17:32.418 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/33' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/17' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/35' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './148/37' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './148' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/19' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/15' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/31' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/36' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/20' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/12' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/39' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/10' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/34' 2024-03-27T06:17:32.419 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/38' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/11' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/16' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/32' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/30' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/14' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/18' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/13' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/33' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/17' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/35' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './149/37' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './149' 2024-03-27T06:17:32.420 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/19' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/15' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/31' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/36' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/20' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/12' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/39' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/10' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/34' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/38' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/11' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/16' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/32' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/30' 2024-03-27T06:17:32.421 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/14' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/18' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/13' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/33' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/17' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/35' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './150/37' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './150' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/19' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/15' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/31' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/36' 2024-03-27T06:17:32.422 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/20' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/12' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/39' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/10' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/34' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/38' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/11' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/16' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/32' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/30' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/14' 2024-03-27T06:17:32.423 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/18' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/13' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/33' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/17' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/35' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './151/37' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './151' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/19' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/15' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/31' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/36' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/20' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/12' 2024-03-27T06:17:32.424 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/39' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/10' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/34' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/38' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/11' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/16' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/32' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/30' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/14' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/18' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/13' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/33' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/17' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/35' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed './152/37' 2024-03-27T06:17:32.425 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './152' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/19' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/15' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/31' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/36' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/20' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/12' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/39' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/10' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/34' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/38' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/11' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/16' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/32' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/30' 2024-03-27T06:17:32.426 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/14' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/18' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/13' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/33' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/17' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/35' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './153/37' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './153' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/19' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/15' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/31' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/36' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/20' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/12' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/39' 2024-03-27T06:17:32.427 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/10' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/34' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/38' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/11' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/16' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/32' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/30' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/14' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/18' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/13' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/33' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/17' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/35' 2024-03-27T06:17:32.428 INFO:tasks.workunit.client.0.smithi132.stdout:removed './154/37' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './154' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/19' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/15' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/31' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/36' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/20' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/12' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/39' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/10' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/34' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/38' 2024-03-27T06:17:32.429 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/11' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/16' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/32' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/30' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/14' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/18' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/13' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/33' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/17' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/35' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './155/37' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './155' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/19' 2024-03-27T06:17:32.430 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/15' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/31' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/36' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/20' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/12' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/39' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/10' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/34' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/38' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/11' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/16' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/32' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/30' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/14' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/18' 2024-03-27T06:17:32.431 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/13' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/33' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/17' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/35' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './156/37' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './156' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/19' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/15' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/31' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/36' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/20' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/12' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/39' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/10' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/34' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/38' 2024-03-27T06:17:32.432 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/11' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/16' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/32' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/30' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/14' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/18' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/13' 2024-03-27T06:17:32.433 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/33' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/17' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/35' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed './157/37' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './157' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/19' 2024-03-27T06:17:33.248 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/15' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/31' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/36' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/20' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/12' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/39' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/10' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/34' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/38' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/11' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/16' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/32' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/30' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/14' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/18' 2024-03-27T06:17:33.249 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/13' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/33' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/17' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/35' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './158/37' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './158' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/19' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/15' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/31' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/36' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/20' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/12' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/39' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/10' 2024-03-27T06:17:33.250 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/34' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/38' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/11' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/16' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/32' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/30' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/14' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/18' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/13' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/33' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/17' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/35' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './159/37' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './159' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/19' 2024-03-27T06:17:33.251 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/15' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/31' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/36' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/20' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/12' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/39' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/10' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/34' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/38' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/11' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/16' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/32' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/30' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/14' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/18' 2024-03-27T06:17:33.252 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/13' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/33' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/17' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/35' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './160/37' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './160' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/19' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/15' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/31' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/36' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/20' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/12' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/39' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/10' 2024-03-27T06:17:33.253 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/34' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/38' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/11' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/16' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/32' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/30' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/14' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/18' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/13' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/33' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/17' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/35' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './161/37' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './161' 2024-03-27T06:17:33.254 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/19' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/15' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/31' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/36' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/20' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/12' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/39' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/10' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/34' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/38' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/11' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/16' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/32' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/30' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/14' 2024-03-27T06:17:33.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/18' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/13' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/33' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/17' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/35' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './162/37' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './162' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/19' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/15' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/31' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/36' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/20' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/12' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/39' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/10' 2024-03-27T06:17:33.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/34' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/38' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/11' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/16' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/32' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/30' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/14' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/18' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/13' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/33' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/17' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/35' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './163/37' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './163' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/19' 2024-03-27T06:17:33.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/15' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/31' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/36' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/20' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/12' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/39' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/10' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/34' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/38' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/11' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/16' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/32' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/30' 2024-03-27T06:17:33.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/14' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/18' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/13' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/33' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/17' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/35' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './164/37' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './164' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/19' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/15' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/31' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/36' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/20' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/12' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/39' 2024-03-27T06:17:33.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/10' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/34' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/38' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/11' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/16' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/32' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/30' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/14' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/18' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/13' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/33' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/17' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/35' 2024-03-27T06:17:33.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './165/37' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './165' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/19' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/15' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/31' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/36' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/20' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/12' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/39' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/10' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/34' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/38' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/11' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/16' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/32' 2024-03-27T06:17:33.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/30' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/14' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/18' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/13' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/33' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/17' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/35' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './166/37' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './166' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/19' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/15' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/31' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/36' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/20' 2024-03-27T06:17:33.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/12' 2024-03-27T06:17:33.263 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/39' 2024-03-27T06:17:33.263 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/10' 2024-03-27T06:17:33.263 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/34' 2024-03-27T06:17:33.263 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/38' 2024-03-27T06:17:33.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:17:33.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:17:33.836 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/11' 2024-03-27T06:17:33.836 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/16' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/32' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/30' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/14' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/18' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/13' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/33' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/17' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/35' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './167/37' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './167' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/19' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/15' 2024-03-27T06:17:33.837 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/31' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/36' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/20' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/12' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/39' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/10' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/34' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/38' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/11' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/16' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/32' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/30' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/14' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/18' 2024-03-27T06:17:33.838 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/13' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/33' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/17' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/35' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './168/37' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './168' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/19' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/15' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/31' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/36' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/20' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/12' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/39' 2024-03-27T06:17:33.839 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/10' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/34' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/38' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/11' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/16' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/32' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/30' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/14' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/18' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/13' 2024-03-27T06:17:33.840 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/33' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/17' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/35' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './169/37' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './169' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/19' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/15' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/31' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/36' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/20' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/12' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/39' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/10' 2024-03-27T06:17:33.841 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/34' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/38' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/11' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/16' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/32' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/30' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/14' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/18' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/13' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/33' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/17' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/35' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed './170/37' 2024-03-27T06:17:33.842 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './170' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/19' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/15' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/31' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/36' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/20' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/12' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/39' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/10' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/34' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/38' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/11' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/16' 2024-03-27T06:17:33.843 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/32' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/30' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/14' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/18' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/13' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/33' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/17' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/35' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './171/37' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './171' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/19' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/15' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/31' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/36' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/20' 2024-03-27T06:17:33.844 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/12' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/39' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/10' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/34' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/38' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/11' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/16' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/32' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/30' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/14' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/18' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/13' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/33' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/17' 2024-03-27T06:17:33.845 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/35' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './172/37' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './172' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/19' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/15' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/31' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/36' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/20' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/12' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/39' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/10' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/34' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/38' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/11' 2024-03-27T06:17:33.846 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/16' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/32' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/30' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/14' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/18' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/13' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/33' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/17' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/35' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './173/37' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './173' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/19' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/15' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/31' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/36' 2024-03-27T06:17:33.847 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/20' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/12' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/39' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/10' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/34' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/38' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/11' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/16' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/32' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/30' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/14' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/18' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/13' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/33' 2024-03-27T06:17:33.848 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/17' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/35' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './174/37' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './174' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/19' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/15' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/31' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/36' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/20' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/12' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/39' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/10' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/34' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/38' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/11' 2024-03-27T06:17:33.849 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/16' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/32' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/30' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/14' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/18' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/13' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/33' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/17' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/35' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './175/37' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './175' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/19' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/15' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/31' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/36' 2024-03-27T06:17:33.850 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/20' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/12' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/39' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/10' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/34' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/38' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/11' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/16' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/32' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/30' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/14' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/18' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/13' 2024-03-27T06:17:33.851 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/33' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/17' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/35' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed './176/37' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './176' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/19' 2024-03-27T06:17:33.852 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/15' 2024-03-27T06:17:34.454 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/31' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/36' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/20' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/12' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/39' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/10' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/34' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/38' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/11' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/16' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/32' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/30' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/14' 2024-03-27T06:17:34.455 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/18' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/13' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/33' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/17' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/35' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './177/37' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './177' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/19' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/15' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/31' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/36' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/20' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/12' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/39' 2024-03-27T06:17:34.456 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/10' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/34' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/38' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/11' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/16' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/32' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/30' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/14' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/18' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/13' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/33' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/17' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/35' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './178/37' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './178' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/19' 2024-03-27T06:17:34.457 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/15' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/31' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/36' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/20' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/12' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/39' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/10' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/34' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/38' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/11' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/16' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/32' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/30' 2024-03-27T06:17:34.458 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/14' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/18' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/13' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/33' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/17' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/35' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './179/37' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './179' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/19' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/15' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/31' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/36' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/20' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/12' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/39' 2024-03-27T06:17:34.459 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/10' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/34' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/38' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/11' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/16' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/32' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/30' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/14' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/18' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/13' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/33' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/17' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/35' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './180/37' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './180' 2024-03-27T06:17:34.460 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/19' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/15' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/31' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/36' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/20' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/12' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/39' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/10' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/34' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/38' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/11' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/16' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/32' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/30' 2024-03-27T06:17:34.461 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/14' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/18' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/13' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/33' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/17' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/35' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './181/37' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './181' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/19' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/15' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/31' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/36' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/20' 2024-03-27T06:17:34.462 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/12' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/39' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/10' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/34' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/38' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/11' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/16' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/32' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/30' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/14' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/18' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/13' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/33' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/17' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/35' 2024-03-27T06:17:34.463 INFO:tasks.workunit.client.0.smithi132.stdout:removed './182/37' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './182' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/19' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/15' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/31' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/36' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/20' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/12' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/39' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/10' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/34' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/38' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/11' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/16' 2024-03-27T06:17:34.464 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/32' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/30' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/14' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/18' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/13' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/33' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/17' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/35' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './183/37' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './183' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/19' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/15' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/31' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/36' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/20' 2024-03-27T06:17:34.465 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/12' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/39' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/10' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/34' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/38' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/11' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/16' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/32' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/30' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/14' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/18' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/13' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/33' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/17' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/35' 2024-03-27T06:17:34.466 INFO:tasks.workunit.client.0.smithi132.stdout:removed './184/37' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './184' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/19' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/15' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/31' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/36' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/20' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/12' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/39' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/10' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/34' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/38' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/11' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/16' 2024-03-27T06:17:34.467 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/32' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/30' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/14' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/18' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/13' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/33' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/17' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/35' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './185/37' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './185' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/19' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/15' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/31' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/36' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/20' 2024-03-27T06:17:34.468 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/12' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/39' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/10' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/34' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/38' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/11' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/16' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/32' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/30' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/14' 2024-03-27T06:17:34.469 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/18' 2024-03-27T06:17:35.144 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/13' 2024-03-27T06:17:35.144 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/33' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/17' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/35' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './186/37' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './186' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/19' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/15' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/31' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/36' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/20' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/12' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/39' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/10' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/34' 2024-03-27T06:17:35.145 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/38' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/11' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/16' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/32' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/30' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/14' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/18' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/13' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/33' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/17' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/35' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './187/37' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './187' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/19' 2024-03-27T06:17:35.146 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/15' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/31' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/36' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/20' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/12' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/39' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/10' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/34' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/38' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/11' 2024-03-27T06:17:35.147 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/16' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/32' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/30' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/14' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/18' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/13' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/33' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/17' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/35' 2024-03-27T06:17:35.148 INFO:tasks.workunit.client.0.smithi132.stdout:removed './188/37' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './188' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/19' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/15' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/31' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/36' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/20' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/12' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/39' 2024-03-27T06:17:35.149 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/10' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/34' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/38' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/11' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/16' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/32' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/30' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/14' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/18' 2024-03-27T06:17:35.150 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/13' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/33' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/17' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/35' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './189/37' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './189' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/19' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/15' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/31' 2024-03-27T06:17:35.151 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/36' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/20' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/12' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/39' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/10' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/34' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/38' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/11' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/16' 2024-03-27T06:17:35.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/32' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/30' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/14' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/18' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/13' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/33' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/17' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/35' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './190/37' 2024-03-27T06:17:35.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './190' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/19' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/15' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/31' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/36' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/20' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/12' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/39' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/10' 2024-03-27T06:17:35.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/34' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/38' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/11' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/16' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/32' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/30' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/14' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/18' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/13' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/33' 2024-03-27T06:17:35.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/17' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/35' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './191/37' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './191' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/19' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/15' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/31' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/36' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/20' 2024-03-27T06:17:35.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/12' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/39' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/10' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/34' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/38' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/11' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/16' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/32' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/30' 2024-03-27T06:17:35.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/14' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/18' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/13' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/33' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/17' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/35' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './192/37' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './192' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/19' 2024-03-27T06:17:35.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/15' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/31' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/36' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/20' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/12' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/39' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/10' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/34' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/38' 2024-03-27T06:17:35.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/11' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/16' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/32' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/30' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/14' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/18' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/13' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/33' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/17' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/35' 2024-03-27T06:17:35.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './193/37' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './193' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/19' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/15' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/31' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/36' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/20' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/12' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/39' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/10' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/34' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/38' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/11' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/16' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/32' 2024-03-27T06:17:35.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/30' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/14' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/18' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/13' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/33' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/17' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/35' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './194/37' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './194' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/19' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/15' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/31' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/36' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/20' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/12' 2024-03-27T06:17:35.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/39' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/10' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/34' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/38' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/11' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/16' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/32' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/30' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/14' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/18' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/13' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/33' 2024-03-27T06:17:35.163 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/17' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/35' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './195/37' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './195' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/19' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/15' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/31' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/36' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/20' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/12' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/39' 2024-03-27T06:17:35.164 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/10' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/34' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/38' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/11' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/16' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/32' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/30' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/14' 2024-03-27T06:17:35.763 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/18' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/13' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/33' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/17' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/35' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './196/37' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './196' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/19' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/15' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/31' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/36' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/20' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/12' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/39' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/10' 2024-03-27T06:17:35.764 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/34' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/38' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/11' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/16' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/32' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/30' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/14' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/18' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/13' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/33' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/17' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/35' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './197/37' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './197' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/19' 2024-03-27T06:17:35.765 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/15' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/31' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/36' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/20' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/12' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/39' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/10' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/34' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/38' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/11' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/16' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/32' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/30' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/14' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/18' 2024-03-27T06:17:35.766 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/13' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/33' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/17' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/35' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './198/37' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './198' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/19' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/15' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/31' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/36' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/20' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/12' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/39' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/10' 2024-03-27T06:17:35.767 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/34' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/38' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/11' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/16' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/32' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/30' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/14' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/18' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/13' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/33' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/17' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/35' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './199/37' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './199' 2024-03-27T06:17:35.768 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/21' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/27' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/31' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/36' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/20' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/39' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/34' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/38' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/28' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/24' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/29' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/32' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/22' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/25' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/30' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/33' 2024-03-27T06:17:35.769 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/23' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/35' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/26' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './200/37' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './200' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/21' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/27' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/31' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/36' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/20' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/39' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/34' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/38' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/28' 2024-03-27T06:17:35.770 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/24' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/29' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/32' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/22' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/25' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/30' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/33' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/23' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/35' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/26' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './201/37' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './201' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/21' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/27' 2024-03-27T06:17:35.771 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/31' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/36' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/20' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/39' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/34' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/38' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/28' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/24' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/29' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/32' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/22' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/25' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/30' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/33' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/23' 2024-03-27T06:17:35.772 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/35' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/26' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './202/37' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './202' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/21' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/27' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/31' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/36' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/20' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/39' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/34' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/38' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/28' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/24' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/29' 2024-03-27T06:17:35.773 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/32' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/22' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/25' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/30' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/33' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/23' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/35' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/26' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './203/37' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './203' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/21' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/27' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/31' 2024-03-27T06:17:35.774 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/36' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/20' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/39' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/34' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/38' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/28' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/24' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/29' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/32' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/22' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/25' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/30' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/33' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/23' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/35' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/26' 2024-03-27T06:17:35.775 INFO:tasks.workunit.client.0.smithi132.stdout:removed './204/37' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './204' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/21' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/27' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/31' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/36' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/20' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/39' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/34' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/38' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/28' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/24' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/29' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/32' 2024-03-27T06:17:35.776 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/22' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/25' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/30' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/33' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/23' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/35' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/26' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './205/37' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './205' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/21' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/27' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/31' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/36' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/20' 2024-03-27T06:17:35.777 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/39' 2024-03-27T06:17:36.374 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/34' 2024-03-27T06:17:36.374 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/38' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/28' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/24' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/29' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/32' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/22' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/25' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/30' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/33' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/23' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/35' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/26' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './206/37' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './206' 2024-03-27T06:17:36.375 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/21' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/27' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/31' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/36' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/20' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/39' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/34' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/38' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/28' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/24' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/29' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/32' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/22' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/25' 2024-03-27T06:17:36.376 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/30' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/33' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/23' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/35' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/26' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './207/37' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './207' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/21' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/27' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/31' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/36' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/20' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/39' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/34' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/38' 2024-03-27T06:17:36.377 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/28' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/24' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/29' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/32' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/22' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/25' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/30' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/33' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/23' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/35' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/26' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './208/37' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './208' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/21' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/27' 2024-03-27T06:17:36.378 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/31' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/36' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/20' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/39' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/34' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/38' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/28' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/24' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/29' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/32' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/22' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/25' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/30' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/33' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/23' 2024-03-27T06:17:36.379 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/35' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/26' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './209/37' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './209' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/21' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/27' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/31' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/36' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/20' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/39' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/34' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/38' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/28' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/24' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/29' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/32' 2024-03-27T06:17:36.380 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/22' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/25' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/30' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/33' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/23' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/35' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/26' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './210/37' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './210' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/21' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/27' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/31' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/36' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/20' 2024-03-27T06:17:36.381 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/39' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/34' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/38' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/28' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/24' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/29' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/32' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/22' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/25' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/30' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/33' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/23' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/35' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/26' 2024-03-27T06:17:36.382 INFO:tasks.workunit.client.0.smithi132.stdout:removed './211/37' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './211' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/21' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/27' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/31' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/36' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/20' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/39' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/34' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/38' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/28' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/24' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/29' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/32' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/22' 2024-03-27T06:17:36.383 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/25' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/30' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/33' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/23' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/35' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/26' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './212/37' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './212' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/21' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/27' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/31' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/36' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/20' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/39' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/34' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/38' 2024-03-27T06:17:36.384 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/28' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/24' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/29' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/32' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/22' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/25' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/30' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/33' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/23' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/35' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/26' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './213/37' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './213' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/21' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/27' 2024-03-27T06:17:36.385 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/31' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/36' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/20' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/39' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/34' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/38' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/28' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/24' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/29' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/32' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/22' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/25' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/30' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/33' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/23' 2024-03-27T06:17:36.386 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/35' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/26' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './214/37' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './214' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/21' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/27' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/31' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/36' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/20' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/39' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/34' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/38' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/28' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/24' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/29' 2024-03-27T06:17:36.387 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/32' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/22' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/25' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/30' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/33' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/23' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/35' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/26' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './215/37' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './215' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/21' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/27' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/31' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/36' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/20' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/39' 2024-03-27T06:17:36.388 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/34' 2024-03-27T06:17:36.389 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/38' 2024-03-27T06:17:37.068 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/28' 2024-03-27T06:17:37.068 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/24' 2024-03-27T06:17:37.068 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/29' 2024-03-27T06:17:37.068 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/32' 2024-03-27T06:17:37.068 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/22' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/25' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/30' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/33' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/23' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/35' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/26' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './216/37' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './216' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/21' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/27' 2024-03-27T06:17:37.069 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/31' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/36' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/20' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/39' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/34' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/38' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/28' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/24' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/29' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/32' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/22' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/25' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/30' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/33' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/23' 2024-03-27T06:17:37.070 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/35' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/26' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './217/37' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './217' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/21' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/27' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/31' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/36' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/20' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/39' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/34' 2024-03-27T06:17:37.071 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/38' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/28' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/24' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/29' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/32' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/22' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/25' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/30' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/33' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/23' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/35' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/26' 2024-03-27T06:17:37.072 INFO:tasks.workunit.client.0.smithi132.stdout:removed './218/37' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './218' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/21' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/27' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/31' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/36' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/20' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/39' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/34' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/38' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/28' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/24' 2024-03-27T06:17:37.073 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/29' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/32' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/22' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/25' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/30' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/33' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/23' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/35' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/26' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './219/37' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './219' 2024-03-27T06:17:37.074 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/21' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/27' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/31' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/36' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/20' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/39' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/34' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/38' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/28' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/24' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/29' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/32' 2024-03-27T06:17:37.075 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/22' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/25' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/30' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/33' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/23' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/35' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/26' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './220/37' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './220' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/21' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/27' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/31' 2024-03-27T06:17:37.076 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/36' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/20' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/39' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/34' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/38' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/28' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/24' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/29' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/32' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/22' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/25' 2024-03-27T06:17:37.077 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/30' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/33' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/23' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/35' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/26' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './221/37' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './221' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/21' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/27' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/31' 2024-03-27T06:17:37.078 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/36' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/20' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/39' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/34' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/38' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/28' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/24' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/29' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/32' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/22' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/25' 2024-03-27T06:17:37.079 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/30' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/33' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/23' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/35' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/26' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './222/37' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './222' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/21' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/27' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/31' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/36' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/20' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/39' 2024-03-27T06:17:37.080 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/34' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/38' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/28' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/24' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/29' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/32' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/22' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/25' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/30' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/33' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/23' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/35' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/26' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed './223/37' 2024-03-27T06:17:37.081 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './223' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/21' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/27' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/31' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/36' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/20' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/39' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/34' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/38' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/28' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/24' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/29' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/32' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/22' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/25' 2024-03-27T06:17:37.082 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/30' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/33' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/23' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/35' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/26' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './224/37' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './224' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/21' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/27' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/31' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/36' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/20' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/39' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/34' 2024-03-27T06:17:37.083 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/38' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/28' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/24' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/29' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/32' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/22' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/25' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/30' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/33' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/23' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/35' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/26' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed './225/37' 2024-03-27T06:17:37.084 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './225' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/21' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/27' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/31' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/36' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/20' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/39' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/34' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/38' 2024-03-27T06:17:37.085 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/28' 2024-03-27T06:17:37.086 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/24' 2024-03-27T06:17:37.636 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/29' 2024-03-27T06:17:37.636 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/32' 2024-03-27T06:17:37.636 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/22' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/25' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/30' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/33' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/23' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/35' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/26' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './226/37' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './226' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/21' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/27' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/31' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/36' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/20' 2024-03-27T06:17:37.637 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/39' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/34' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/38' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/28' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/24' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/29' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/32' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/22' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/25' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/30' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/33' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/23' 2024-03-27T06:17:37.638 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/35' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/26' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './227/37' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './227' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/21' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/27' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/31' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/36' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/20' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/39' 2024-03-27T06:17:37.639 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/34' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/38' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/28' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/24' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/29' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/32' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/22' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/25' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/30' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/33' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/23' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/35' 2024-03-27T06:17:37.640 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/26' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './228/37' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './228' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/21' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/27' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/31' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/36' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/20' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/39' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/34' 2024-03-27T06:17:37.641 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/38' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/28' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/24' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/29' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/32' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/22' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/25' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/30' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/33' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/23' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/35' 2024-03-27T06:17:37.642 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/26' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './229/37' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './229' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/21' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/27' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/31' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/36' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/20' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/39' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/34' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/38' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/28' 2024-03-27T06:17:37.643 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/24' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/29' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/32' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/22' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/25' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/30' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/33' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/23' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/35' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/26' 2024-03-27T06:17:37.644 INFO:tasks.workunit.client.0.smithi132.stdout:removed './230/37' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './230' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/21' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/27' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/31' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/36' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/20' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/39' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/34' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/38' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/28' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/24' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/29' 2024-03-27T06:17:37.645 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/32' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/22' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/25' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/30' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/33' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/23' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/35' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/26' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './231/37' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './231' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/21' 2024-03-27T06:17:37.646 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/27' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/31' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/36' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/20' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/39' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/34' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/38' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/28' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/24' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/29' 2024-03-27T06:17:37.647 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/32' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/22' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/25' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/30' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/33' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/23' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/35' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/26' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './232/37' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './232' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/21' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/27' 2024-03-27T06:17:37.648 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/31' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/36' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/20' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/39' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/34' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/38' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/28' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/24' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/29' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/32' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/22' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/25' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/30' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/33' 2024-03-27T06:17:37.649 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/23' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/35' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/26' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './233/37' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './233' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/21' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/27' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/31' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/36' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/20' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/39' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/34' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/38' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/28' 2024-03-27T06:17:37.650 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/24' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/29' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/32' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/22' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/25' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/30' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/33' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/23' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/35' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/26' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './234/37' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './234' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/21' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/27' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/31' 2024-03-27T06:17:37.651 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/36' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/20' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/39' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/34' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/38' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/28' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/24' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/29' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/32' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/22' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/25' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/30' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/33' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/23' 2024-03-27T06:17:37.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/35' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/26' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './235/37' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './235' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/21' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/27' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/31' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/36' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/20' 2024-03-27T06:17:37.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/39' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/34' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/38' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/28' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/24' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/29' 2024-03-27T06:17:37.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/32' 2024-03-27T06:17:38.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/22' 2024-03-27T06:17:38.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/25' 2024-03-27T06:17:38.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/30' 2024-03-27T06:17:38.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/33' 2024-03-27T06:17:38.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/23' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/35' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/26' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './236/37' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './236' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/21' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/27' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/31' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/36' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/20' 2024-03-27T06:17:38.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/39' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/34' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/38' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/28' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/24' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/29' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/32' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/22' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/25' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/30' 2024-03-27T06:17:38.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/33' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/23' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/35' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/26' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './237/37' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './237' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/21' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/27' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/31' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/36' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/20' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/39' 2024-03-27T06:17:38.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/34' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/38' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/28' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/24' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/29' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/32' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/22' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/25' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/30' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/33' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/23' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/35' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/26' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './238/37' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './238' 2024-03-27T06:17:38.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/21' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/27' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/31' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/36' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/20' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/39' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/34' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/38' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/28' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/24' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/29' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/32' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/22' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/25' 2024-03-27T06:17:38.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/30' 2024-03-27T06:17:38.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/33' 2024-03-27T06:17:38.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/23' 2024-03-27T06:17:38.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/35' 2024-03-27T06:17:38.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/26' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './239/37' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './239' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/21' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/27' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/31' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/36' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/20' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/39' 2024-03-27T06:17:38.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/34' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/38' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/28' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/24' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/29' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/32' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/22' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/25' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/30' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/33' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/23' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/35' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/26' 2024-03-27T06:17:38.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './240/37' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './240' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/21' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/27' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/31' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/36' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/20' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/39' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/34' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/38' 2024-03-27T06:17:38.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/28' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/24' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/29' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/32' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/22' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/25' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/30' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/33' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/23' 2024-03-27T06:17:38.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/35' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/26' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './241/37' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './241' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/21' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/27' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/31' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/36' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/20' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/39' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/34' 2024-03-27T06:17:38.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/38' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/28' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/24' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/29' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/32' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/22' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/25' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/30' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/33' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/23' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/35' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/26' 2024-03-27T06:17:38.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './242/37' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './242' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/21' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/27' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/31' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/36' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/20' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/39' 2024-03-27T06:17:38.316 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/34' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/38' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/28' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/24' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/29' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/32' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/22' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/25' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/30' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/33' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/23' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/35' 2024-03-27T06:17:38.317 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/26' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './243/37' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './243' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/21' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/27' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/31' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/36' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/20' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/39' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/34' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/38' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/28' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/24' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/29' 2024-03-27T06:17:38.318 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/32' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/22' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/25' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/30' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/33' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/23' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/35' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/26' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './244/37' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './244' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/21' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/27' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/31' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/36' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/20' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/39' 2024-03-27T06:17:38.319 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/34' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/38' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/28' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/24' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/29' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/32' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/22' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/25' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/30' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/33' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/23' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/35' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/26' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './245/37' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './245' 2024-03-27T06:17:38.320 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/21' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/27' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/31' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/36' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/20' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/39' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/34' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/38' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/28' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/24' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/29' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/32' 2024-03-27T06:17:38.321 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/22' 2024-03-27T06:17:39.017 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/25' 2024-03-27T06:17:39.017 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/30' 2024-03-27T06:17:39.017 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/33' 2024-03-27T06:17:39.017 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/23' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/35' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/26' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './246/37' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './246' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/21' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/27' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/31' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/36' 2024-03-27T06:17:39.018 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/20' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/39' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/34' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/38' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/28' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/24' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/29' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/32' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/22' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/25' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/30' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/33' 2024-03-27T06:17:39.019 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/23' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/35' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/26' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './247/37' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './247' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/21' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/27' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/31' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/36' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/20' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/39' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/34' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/38' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/28' 2024-03-27T06:17:39.020 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/24' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/29' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/32' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/22' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/25' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/30' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/33' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/23' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/35' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/26' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './248/37' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './248' 2024-03-27T06:17:39.021 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/21' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/27' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/31' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/36' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/20' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/39' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/34' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/38' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/28' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/24' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/29' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/32' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/22' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/25' 2024-03-27T06:17:39.022 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/30' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/33' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/23' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/35' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/26' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './249/37' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './249' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/21' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/27' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/31' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/36' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/20' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/39' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/34' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/38' 2024-03-27T06:17:39.023 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/28' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/24' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/29' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/32' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/22' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/25' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/30' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/33' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/23' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/35' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/26' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './250/37' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './250' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/21' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/27' 2024-03-27T06:17:39.024 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/31' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/36' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/20' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/39' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/34' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/38' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/28' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/24' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/29' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/32' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/22' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/25' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/30' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/33' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/23' 2024-03-27T06:17:39.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/35' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/26' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './251/37' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './251' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/21' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/27' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/31' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/36' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/20' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/39' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/34' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/38' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/28' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/24' 2024-03-27T06:17:39.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/29' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/32' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/22' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/25' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/30' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/33' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/23' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/35' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/26' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './252/37' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './252' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/21' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/27' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/31' 2024-03-27T06:17:39.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/36' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/20' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/39' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/34' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/38' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/28' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/24' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/29' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/32' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/22' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/25' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/30' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/33' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/23' 2024-03-27T06:17:39.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/35' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/26' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './253/37' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './253' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/21' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/27' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/31' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/36' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/20' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/39' 2024-03-27T06:17:39.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/34' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/38' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/28' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/24' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/29' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/32' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/22' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/25' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/30' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/33' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/23' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/35' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/26' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './254/37' 2024-03-27T06:17:39.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './254' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/21' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/27' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/31' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/36' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/20' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/39' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/34' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/38' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/28' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/24' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/29' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/32' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/22' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/25' 2024-03-27T06:17:39.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/30' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/33' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/23' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/35' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/26' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './255/37' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './255' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/21' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/27' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/31' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/36' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/20' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/39' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/34' 2024-03-27T06:17:39.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/38' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/28' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/24' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/29' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/32' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/22' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/25' 2024-03-27T06:17:39.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/30' 2024-03-27T06:17:39.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/33' 2024-03-27T06:17:39.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/23' 2024-03-27T06:17:39.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/35' 2024-03-27T06:17:39.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/26' 2024-03-27T06:17:39.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './256/37' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './256' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/21' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/27' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/31' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/36' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/20' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/39' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/34' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/38' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/28' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/24' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/29' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/32' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/22' 2024-03-27T06:17:39.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/25' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/30' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/33' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/23' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/35' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/26' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './257/37' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './257' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/21' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/27' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/31' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/36' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/20' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/39' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/34' 2024-03-27T06:17:39.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/38' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/28' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/24' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/29' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/32' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/22' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/25' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/30' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/33' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/23' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/35' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/26' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './258/37' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './258' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/21' 2024-03-27T06:17:39.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/27' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/31' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/36' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/20' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/39' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/34' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/38' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/28' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/24' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/29' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/32' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/22' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/25' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/30' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/33' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/23' 2024-03-27T06:17:39.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/35' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/26' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './259/37' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './259' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/21' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/27' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/31' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/36' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/20' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/39' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/34' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/38' 2024-03-27T06:17:39.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/28' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/24' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/29' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/32' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/22' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/25' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/30' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/33' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/23' 2024-03-27T06:17:39.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/35' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/26' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './260/37' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './260' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/21' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/27' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/31' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/36' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/20' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/39' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/34' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/38' 2024-03-27T06:17:39.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/28' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/24' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/29' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/32' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/22' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/25' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/30' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/33' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/23' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/35' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/26' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './261/37' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './261' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/21' 2024-03-27T06:17:39.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/27' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/31' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/36' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/20' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/39' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/34' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/38' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/28' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/24' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/29' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/32' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/22' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/25' 2024-03-27T06:17:39.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/30' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/33' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/23' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/35' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/26' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './262/37' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './262' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/21' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/27' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/31' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/36' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/20' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/39' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/34' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/38' 2024-03-27T06:17:39.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/28' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/24' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/29' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/32' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/22' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/25' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/30' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/33' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/23' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/35' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/26' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './263/37' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './263' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/21' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/27' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/31' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/36' 2024-03-27T06:17:39.667 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/20' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/39' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/34' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/38' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/28' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/24' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/29' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/32' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/22' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/25' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/30' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/33' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/23' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/35' 2024-03-27T06:17:39.668 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/26' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './264/37' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './264' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/21' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/27' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/31' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/36' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/20' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/39' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/34' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/38' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/28' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/24' 2024-03-27T06:17:39.669 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/29' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/32' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/22' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/25' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/30' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/33' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/23' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/35' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/26' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './265/37' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './265' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/21' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/27' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/31' 2024-03-27T06:17:39.670 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/36' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/20' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/39' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/34' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/38' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/28' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/24' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/29' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/32' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/22' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/25' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/30' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/33' 2024-03-27T06:17:39.671 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/23' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/35' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/26' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './266/37' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './266' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/21' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/27' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/31' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/36' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/20' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/39' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/34' 2024-03-27T06:17:40.301 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/38' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/28' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/24' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/29' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/32' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/22' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/25' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/30' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/33' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/23' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/35' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/26' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed './267/37' 2024-03-27T06:17:40.302 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './267' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/21' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/27' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/31' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/36' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/20' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/39' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/34' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/38' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/28' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/24' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/29' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/32' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/22' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/25' 2024-03-27T06:17:40.303 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/30' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/33' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/23' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/35' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/26' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './268/37' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './268' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/21' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/27' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/31' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/36' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/20' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/39' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/34' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/38' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/28' 2024-03-27T06:17:40.304 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/24' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/29' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/32' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/22' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/25' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/30' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/33' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/23' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/35' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/26' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './269/37' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './269' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/21' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/27' 2024-03-27T06:17:40.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/31' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/36' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/20' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/39' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/34' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/38' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/28' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/24' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/29' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/32' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/22' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/25' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/30' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/33' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/23' 2024-03-27T06:17:40.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/35' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/26' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './270/37' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './270' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/21' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/27' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/31' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/36' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/20' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/39' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/34' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/38' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/28' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/24' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/29' 2024-03-27T06:17:40.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/32' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/22' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/25' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/30' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/33' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/23' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/35' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/26' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './271/37' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './271' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/21' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/27' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/31' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/36' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/20' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/39' 2024-03-27T06:17:40.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/34' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/38' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/28' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/24' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/29' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/32' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/22' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/25' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/30' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/33' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/23' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/35' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/26' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './272/37' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './272' 2024-03-27T06:17:40.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/21' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/27' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/31' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/36' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/20' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/39' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/34' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/38' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/28' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/24' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/29' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/32' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/22' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/25' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/30' 2024-03-27T06:17:40.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/33' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/23' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/35' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/26' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './273/37' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './273' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/21' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/27' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/31' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/36' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/20' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/39' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/34' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/38' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/28' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/24' 2024-03-27T06:17:40.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/29' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/32' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/22' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/25' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/30' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/33' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/23' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/35' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/26' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './274/37' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './274' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/21' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/27' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/31' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/36' 2024-03-27T06:17:40.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/20' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/39' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/34' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/38' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/28' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/24' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/29' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/32' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/22' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/25' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/30' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/33' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/23' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/35' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/26' 2024-03-27T06:17:40.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './275/37' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './275' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/21' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/27' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/31' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/36' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/20' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/39' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/34' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/38' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/28' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/24' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/29' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/32' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/22' 2024-03-27T06:17:40.314 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/25' 2024-03-27T06:17:40.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/30' 2024-03-27T06:17:40.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/33' 2024-03-27T06:17:40.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/23' 2024-03-27T06:17:40.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/35' 2024-03-27T06:17:40.315 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/26' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './276/37' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './276' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/21' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/27' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/31' 2024-03-27T06:17:41.025 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/36' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/20' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/39' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/34' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/38' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/28' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/24' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/29' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/32' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/22' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/25' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/30' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/33' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/23' 2024-03-27T06:17:41.026 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/35' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/26' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './277/37' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './277' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/21' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/27' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/31' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/36' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/20' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/39' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/34' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/38' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/28' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/24' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/29' 2024-03-27T06:17:41.027 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/32' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/22' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/25' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/30' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/33' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/23' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/35' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/26' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './278/37' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './278' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/21' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/27' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/31' 2024-03-27T06:17:41.028 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/36' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/20' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/39' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/34' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/38' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/28' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/24' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/29' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/32' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/22' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/25' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/30' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/33' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/23' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/35' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/26' 2024-03-27T06:17:41.029 INFO:tasks.workunit.client.0.smithi132.stdout:removed './279/37' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './279' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/21' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/27' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/31' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/36' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/20' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/39' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/34' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/38' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/28' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/24' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/29' 2024-03-27T06:17:41.030 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/32' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/22' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/25' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/30' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/33' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/23' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/35' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/26' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './280/37' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './280' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/21' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/27' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/31' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/36' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/20' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/39' 2024-03-27T06:17:41.031 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/34' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/38' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/28' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/24' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/29' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/32' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/22' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/25' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/30' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/33' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/23' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/35' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/26' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './281/37' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './281' 2024-03-27T06:17:41.032 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/21' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/27' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/31' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/36' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/20' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/39' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/34' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/38' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/28' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/24' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/29' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/32' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/22' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/25' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/30' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/33' 2024-03-27T06:17:41.033 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/23' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/35' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/26' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './282/37' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './282' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/21' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/27' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/31' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/36' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/20' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/39' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/34' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/38' 2024-03-27T06:17:41.034 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/28' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/24' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/29' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/32' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/22' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/25' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/30' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/33' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/23' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/35' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/26' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './283/37' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './283' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/21' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/27' 2024-03-27T06:17:41.035 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/31' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/36' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/20' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/39' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/34' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/38' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/28' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/24' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/29' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/32' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/22' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/25' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/30' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/33' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/23' 2024-03-27T06:17:41.036 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/35' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/26' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './284/37' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './284' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/21' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/27' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/31' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/36' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/20' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/39' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/34' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/38' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/28' 2024-03-27T06:17:41.037 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/24' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/29' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/32' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/22' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/25' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/30' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/33' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/23' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/35' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/26' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './285/37' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './285' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/21' 2024-03-27T06:17:41.038 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/27' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/31' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/36' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/20' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/39' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/34' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/38' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/28' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/24' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/29' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/32' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/22' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/25' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/30' 2024-03-27T06:17:41.039 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/33' 2024-03-27T06:17:41.040 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/23' 2024-03-27T06:17:41.040 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/35' 2024-03-27T06:17:41.040 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/26' 2024-03-27T06:17:41.040 INFO:tasks.workunit.client.0.smithi132.stdout:removed './286/37' 2024-03-27T06:17:41.040 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './286' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/21' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/27' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/31' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/36' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/20' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/39' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/34' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/38' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/28' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/24' 2024-03-27T06:17:41.652 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/29' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/32' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/22' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/25' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/30' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/33' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/23' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/35' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/26' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './287/37' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './287' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/21' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/27' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/31' 2024-03-27T06:17:41.653 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/36' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/20' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/39' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/34' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/38' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/28' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/24' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/29' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/32' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/22' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/25' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/30' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/33' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/23' 2024-03-27T06:17:41.654 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/35' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/26' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './288/37' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './288' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/21' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/27' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/31' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/36' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/20' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/39' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/34' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/38' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/28' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/24' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/29' 2024-03-27T06:17:41.655 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/32' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/22' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/25' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/30' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/33' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/23' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/35' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/26' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './289/37' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './289' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/21' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/27' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/31' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/36' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/20' 2024-03-27T06:17:41.656 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/39' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/34' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/38' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/28' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/24' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/29' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/32' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/22' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/25' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/30' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/33' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/23' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/35' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/26' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './290/37' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './290' 2024-03-27T06:17:41.657 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/21' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/27' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/31' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/36' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/20' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/39' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/34' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/38' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/28' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/24' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/29' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/32' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/22' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/25' 2024-03-27T06:17:41.658 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/30' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/33' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/23' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/35' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/26' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './291/37' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './291' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/21' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/27' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/31' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/36' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/20' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/39' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/34' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/38' 2024-03-27T06:17:41.659 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/28' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/24' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/29' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/32' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/22' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/25' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/30' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/33' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/23' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/35' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/26' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './292/37' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './292' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/21' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/27' 2024-03-27T06:17:41.660 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/31' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/36' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/20' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/39' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/34' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/38' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/28' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/24' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/29' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/32' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/22' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/25' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/30' 2024-03-27T06:17:41.661 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/33' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/23' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/35' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/26' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './293/37' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './293' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/21' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/27' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/31' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/36' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/20' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/39' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/34' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/38' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/28' 2024-03-27T06:17:41.662 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/24' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/29' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/32' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/22' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/25' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/30' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/33' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/23' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/35' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/26' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './294/37' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './294' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/21' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/27' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/31' 2024-03-27T06:17:41.663 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/36' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/20' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/39' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/34' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/38' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/28' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/24' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/29' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/32' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/22' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/25' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/30' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/33' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/23' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/35' 2024-03-27T06:17:41.664 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/26' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './295/37' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './295' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/21' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/27' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/31' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/36' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/20' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/39' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/34' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/38' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/28' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/24' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/29' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/32' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/22' 2024-03-27T06:17:41.665 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/25' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/30' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/33' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/23' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/35' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/26' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './296/37' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './296' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/21' 2024-03-27T06:17:41.666 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/27' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/31' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/36' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/20' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/39' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/34' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/38' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/28' 2024-03-27T06:17:42.152 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/24' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/29' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/32' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/22' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/25' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/30' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/33' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/23' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/35' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/26' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './297/37' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './297' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/21' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/27' 2024-03-27T06:17:42.153 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/31' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/36' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/20' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/39' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/34' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/38' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/28' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/24' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/29' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/32' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/22' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/25' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/30' 2024-03-27T06:17:42.154 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/33' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/23' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/35' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/26' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './298/37' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './298' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/21' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/27' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/31' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/36' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/20' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/39' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/34' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/38' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/28' 2024-03-27T06:17:42.155 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/24' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/29' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/32' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/22' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/25' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/30' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/33' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/23' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/35' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/26' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './299/37' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './299' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/19' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/15' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/20' 2024-03-27T06:17:42.156 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/12' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/10' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/11' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/16' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/14' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/18' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/13' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/166/17' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/166' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/19' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/15' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/20' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/12' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/10' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/11' 2024-03-27T06:17:42.157 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/16' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/14' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/18' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/13' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/147/17' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/147' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/19' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/15' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/20' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/12' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/10' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/11' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/16' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/14' 2024-03-27T06:17:42.158 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/18' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/13' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/151/17' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/151' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/19' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/15' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/20' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/12' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/10' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/11' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/16' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/14' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/18' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/13' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/170/17' 2024-03-27T06:17:42.159 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/170' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/19' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/15' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/20' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/12' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/10' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/11' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/16' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/14' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/18' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/13' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/167/17' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/167' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/19' 2024-03-27T06:17:42.160 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/15' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/20' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/12' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/10' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/11' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/16' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/14' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/18' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/13' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/140/17' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/140' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/19' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/15' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/20' 2024-03-27T06:17:42.161 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/12' 2024-03-27T06:17:42.162 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/10' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/11' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/16' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/14' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/18' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/13' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/142/17' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/142' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/19' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/15' 2024-03-27T06:17:42.558 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/20' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/12' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/10' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/11' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/16' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/14' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/18' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/13' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/150/17' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/150' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/19' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/15' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/20' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/12' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/10' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/11' 2024-03-27T06:17:42.559 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/16' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/14' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/18' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/13' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/105/17' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/105' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/19' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/15' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/20' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/12' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/10' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/11' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/16' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/14' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/18' 2024-03-27T06:17:42.560 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/13' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/149/17' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/149' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/19' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/15' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/20' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/12' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/10' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/11' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/16' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/14' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/18' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/13' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/109/17' 2024-03-27T06:17:42.561 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/109' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/19' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/15' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/20' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/12' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/10' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/11' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/16' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/14' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/18' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/13' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/158/17' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/158' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/19' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/15' 2024-03-27T06:17:42.562 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/20' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/12' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/10' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/11' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/16' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/14' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/18' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/13' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/106/17' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/106' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/19' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/15' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/20' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/12' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/10' 2024-03-27T06:17:42.563 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/11' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/16' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/14' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/18' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/13' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/189/17' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/189' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/19' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/15' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/20' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/12' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/10' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/11' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/16' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/14' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/18' 2024-03-27T06:17:42.564 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/13' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/133/17' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/133' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/19' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/15' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/20' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/12' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/10' 2024-03-27T06:17:42.565 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/11' 2024-03-27T06:17:42.952 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/16' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/14' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/18' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/13' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/173/17' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/173' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/19' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/15' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/20' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/12' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/10' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/11' 2024-03-27T06:17:42.953 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/16' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/14' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/18' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/13' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/174/17' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/174' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/19' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/15' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/20' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/12' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/10' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/11' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/16' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/14' 2024-03-27T06:17:42.954 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/18' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/13' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/155/17' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/155' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/19' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/15' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/20' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/12' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/10' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/11' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/16' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/14' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/18' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/13' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/114/17' 2024-03-27T06:17:42.955 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/114' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/19' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/15' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/20' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/12' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/10' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/11' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/16' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/14' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/18' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/13' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/159/17' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/159' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/19' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/15' 2024-03-27T06:17:42.956 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/20' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/12' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/10' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/11' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/16' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/14' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/18' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/13' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/187/17' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/187' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/19' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/15' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/20' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/12' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/10' 2024-03-27T06:17:42.957 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/11' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/16' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/14' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/18' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/13' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/185/17' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/185' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/19' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/15' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/20' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/12' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/10' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/11' 2024-03-27T06:17:42.958 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/16' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/14' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/18' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/13' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/120/17' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/120' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/19' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/15' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/20' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/12' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/10' 2024-03-27T06:17:42.959 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/11' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/16' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/14' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/18' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/13' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/110/17' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/110' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/19' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/15' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/20' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/12' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/10' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/11' 2024-03-27T06:17:42.960 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/16' 2024-03-27T06:17:43.305 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/14' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/18' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/13' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/137/17' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/137' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/19' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/15' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/20' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/12' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/10' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/11' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/16' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/14' 2024-03-27T06:17:43.306 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/18' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/13' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/198/17' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/198' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/19' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/15' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/20' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/12' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/10' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/11' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/16' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/14' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/18' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/13' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/136/17' 2024-03-27T06:17:43.307 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/136' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/19' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/15' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/20' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/12' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/10' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/11' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/16' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/14' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/18' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/13' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/125/17' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/125' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/19' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/15' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/20' 2024-03-27T06:17:43.308 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/12' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/10' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/11' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/16' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/14' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/18' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/13' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/194/17' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/194' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/19' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/15' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/20' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/12' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/10' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/11' 2024-03-27T06:17:43.309 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/16' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/14' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/18' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/13' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/119/17' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/119' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/19' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/15' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/20' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/12' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/10' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/11' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/16' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/14' 2024-03-27T06:17:43.310 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/18' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/13' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/157/17' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/157' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/19' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/15' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/20' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/12' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/10' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/11' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/16' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/14' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/18' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/13' 2024-03-27T06:17:43.311 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/172/17' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/172' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/19' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/15' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/20' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/12' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/10' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/11' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/16' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/14' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/18' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/13' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/186/17' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/186' 2024-03-27T06:17:43.312 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/19' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/15' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/20' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/12' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/10' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/11' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/16' 2024-03-27T06:17:43.313 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/14' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/18' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/13' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/190/17' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/190' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/19' 2024-03-27T06:17:43.678 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/15' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/20' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/12' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/10' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/11' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/16' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/14' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/18' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/13' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/104/17' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/104' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/19' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/15' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/20' 2024-03-27T06:17:43.679 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/12' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/10' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/11' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/16' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/14' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/18' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/13' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/164/17' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/164' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/19' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/15' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/20' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/12' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/10' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/11' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/16' 2024-03-27T06:17:43.680 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/14' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/18' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/13' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/112/17' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/112' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/19' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/15' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/20' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/12' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/10' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/11' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/16' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/14' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/18' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/13' 2024-03-27T06:17:43.681 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/100/17' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/100' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/19' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/15' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/20' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/12' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/10' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/11' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/16' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/14' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/18' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/13' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/134/17' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/134' 2024-03-27T06:17:43.682 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/19' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/15' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/20' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/12' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/10' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/11' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/16' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/14' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/18' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/13' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/115/17' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/115' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/19' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/15' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/20' 2024-03-27T06:17:43.683 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/12' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/10' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/11' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/16' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/14' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/18' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/13' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/116/17' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/116' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/19' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/15' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/20' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/12' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/10' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/11' 2024-03-27T06:17:43.684 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/16' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/14' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/18' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/13' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/195/17' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/195' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/19' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/15' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/20' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/12' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/10' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/11' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/16' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/14' 2024-03-27T06:17:43.685 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/18' 2024-03-27T06:17:44.111 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/13' 2024-03-27T06:17:44.111 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/138/17' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/138' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/19' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/15' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/20' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/12' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/10' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/11' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/16' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/14' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/18' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/13' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/135/17' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/135' 2024-03-27T06:17:44.112 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/19' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/15' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/20' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/12' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/10' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/11' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/16' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/14' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/18' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/13' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/117/17' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/117' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/19' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/15' 2024-03-27T06:17:44.113 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/20' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/12' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/10' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/11' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/16' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/14' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/18' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/13' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/143/17' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/143' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/19' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/15' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/20' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/12' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/10' 2024-03-27T06:17:44.114 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/11' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/16' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/14' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/18' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/13' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/180/17' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/180' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/19' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/15' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/20' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/12' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/10' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/11' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/16' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/14' 2024-03-27T06:17:44.115 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/18' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/13' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/126/17' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/126' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/19' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/15' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/20' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/12' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/10' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/11' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/16' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/14' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/18' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/13' 2024-03-27T06:17:44.116 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/161/17' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/161' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/19' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/15' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/20' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/12' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/10' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/11' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/16' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/14' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/18' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/13' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/176/17' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/176' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/19' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/15' 2024-03-27T06:17:44.117 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/20' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/12' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/10' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/11' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/16' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/14' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/18' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/13' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/111/17' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/111' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/19' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/15' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/20' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/12' 2024-03-27T06:17:44.118 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/10' 2024-03-27T06:17:44.119 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/11' 2024-03-27T06:17:44.119 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/16' 2024-03-27T06:17:44.119 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/14' 2024-03-27T06:17:44.119 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/18' 2024-03-27T06:17:44.119 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/13' 2024-03-27T06:17:44.512 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/191/17' 2024-03-27T06:17:44.512 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/191' 2024-03-27T06:17:44.512 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/19' 2024-03-27T06:17:44.512 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/15' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/20' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/12' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/10' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/11' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/16' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/14' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/18' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/13' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/184/17' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/184' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/19' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/15' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/20' 2024-03-27T06:17:44.513 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/12' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/10' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/11' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/16' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/14' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/18' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/13' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/192/17' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/192' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/19' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/15' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/20' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/12' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/10' 2024-03-27T06:17:44.514 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/11' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/16' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/14' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/18' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/13' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/103/17' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/103' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/19' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/15' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/20' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/12' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/10' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/11' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/16' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/14' 2024-03-27T06:17:44.515 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/18' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/13' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/128/17' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/128' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/19' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/15' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/20' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/12' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/10' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/11' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/16' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/14' 2024-03-27T06:17:44.516 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/18' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/13' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/144/17' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/144' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/19' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/15' 2024-03-27T06:17:44.517 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/20' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/12' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/10' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/11' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/16' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/14' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/18' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/13' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/146/17' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/146' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/19' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/15' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/20' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/12' 2024-03-27T06:17:44.518 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/10' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/11' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/16' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/14' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/18' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/13' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/141/17' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/141' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/19' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/15' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/20' 2024-03-27T06:17:44.519 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/12' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/10' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/11' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/16' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/14' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/18' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/13' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/182/17' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/182' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/19' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/15' 2024-03-27T06:17:44.520 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/20' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/12' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/10' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/11' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/16' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/14' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/18' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/13' 2024-03-27T06:17:44.521 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/107/17' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/107' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/19' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/15' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/20' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/12' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/10' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/11' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/16' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/14' 2024-03-27T06:17:44.935 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/18' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/13' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/178/17' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/178' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/19' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/15' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/20' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/12' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/10' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/11' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/16' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/14' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/18' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/13' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/156/17' 2024-03-27T06:17:44.936 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/156' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/19' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/15' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/20' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/12' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/10' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/11' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/16' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/14' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/18' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/13' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/131/17' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/131' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/19' 2024-03-27T06:17:44.937 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/15' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/20' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/12' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/10' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/11' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/16' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/14' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/18' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/13' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/123/17' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/123' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/19' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/15' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/20' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/12' 2024-03-27T06:17:44.938 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/10' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/11' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/16' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/14' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/18' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/13' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/160/17' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/160' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/19' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/15' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/20' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/12' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/10' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/11' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/16' 2024-03-27T06:17:44.939 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/14' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/18' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/13' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/165/17' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/165' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/19' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/15' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/20' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/12' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/10' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/11' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/16' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/14' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/18' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/13' 2024-03-27T06:17:44.940 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/101/17' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/101' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/19' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/15' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/20' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/12' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/10' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/11' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/16' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/14' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/18' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/13' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/148/17' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/148' 2024-03-27T06:17:44.941 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/19' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/15' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/20' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/12' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/10' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/11' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/16' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/14' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/18' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/13' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/168/17' 2024-03-27T06:17:44.942 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/168' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/19' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/15' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/20' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/12' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/10' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/11' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/16' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/14' 2024-03-27T06:17:45.255 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/18' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/13' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/179/17' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/179' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/19' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/15' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/20' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/12' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/10' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/11' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/16' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/14' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/18' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/13' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/132/17' 2024-03-27T06:17:45.256 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/132' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/19' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/15' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/20' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/12' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/10' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/11' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/16' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/14' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/18' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/13' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/199/17' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/199' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/19' 2024-03-27T06:17:45.257 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/15' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/20' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/12' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/10' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/11' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/16' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/14' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/18' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/13' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/163/17' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/163' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/19' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/15' 2024-03-27T06:17:45.258 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/20' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/12' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/10' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/11' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/16' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/14' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/18' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/13' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/153/17' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/153' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/19' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/15' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/20' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/12' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/10' 2024-03-27T06:17:45.259 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/11' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/16' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/14' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/18' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/13' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/122/17' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/122' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/19' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/15' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/20' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/12' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/10' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/11' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/16' 2024-03-27T06:17:45.260 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/14' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/18' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/13' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/162/17' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/162' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/19' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/15' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/20' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/12' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/10' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/11' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/16' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/14' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/18' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/13' 2024-03-27T06:17:45.261 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/197/17' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/197' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/19' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/15' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/20' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/12' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/10' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/11' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/16' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/14' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/18' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/13' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/113/17' 2024-03-27T06:17:45.262 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/113' 2024-03-27T06:17:45.263 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/19' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/15' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/20' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/12' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/10' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/11' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/16' 2024-03-27T06:17:45.608 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/14' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/18' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/13' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/152/17' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/152' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/19' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/15' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/20' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/12' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/10' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/11' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/16' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/14' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/18' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/13' 2024-03-27T06:17:45.609 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/121/17' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/121' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/19' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/15' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/20' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/12' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/10' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/11' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/16' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/14' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/18' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/13' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/118/17' 2024-03-27T06:17:45.610 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/118' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/19' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/15' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/20' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/12' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/10' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/11' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/16' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/14' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/18' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/13' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/196/17' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/196' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/19' 2024-03-27T06:17:45.611 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/15' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/20' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/12' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/10' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/11' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/16' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/14' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/18' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/13' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/124/17' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/124' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/19' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/15' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/20' 2024-03-27T06:17:45.612 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/12' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/10' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/11' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/16' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/14' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/18' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/13' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/188/17' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/188' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/19' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/15' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/20' 2024-03-27T06:17:45.613 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/12' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/10' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/11' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/16' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/14' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/18' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/13' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/145/17' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/145' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/19' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/15' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/20' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/12' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/10' 2024-03-27T06:17:45.614 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/11' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/16' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/14' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/18' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/13' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/130/17' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/130' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/19' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/15' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/20' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/12' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/10' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/11' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/16' 2024-03-27T06:17:45.615 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/14' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/18' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/13' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/193/17' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/193' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/19' 2024-03-27T06:17:45.616 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/15' 2024-03-27T06:17:45.967 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/20' 2024-03-27T06:17:45.967 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/12' 2024-03-27T06:17:45.967 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/10' 2024-03-27T06:17:45.967 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/11' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/16' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/14' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/18' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/13' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/175/17' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/175' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/19' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/15' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/20' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/12' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/10' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/11' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/16' 2024-03-27T06:17:45.968 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/14' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/18' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/13' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/181/17' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/181' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/19' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/15' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/20' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/12' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/10' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/11' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/16' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/14' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/18' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/13' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/127/17' 2024-03-27T06:17:45.969 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/127' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/19' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/15' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/20' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/12' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/10' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/11' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/16' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/14' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/18' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/13' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/108/17' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/108' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/19' 2024-03-27T06:17:45.970 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/15' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/20' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/12' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/10' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/11' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/16' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/14' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/18' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/13' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/183/17' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/183' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/19' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/15' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/20' 2024-03-27T06:17:45.971 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/12' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/10' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/11' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/16' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/14' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/18' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/13' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/177/17' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/177' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/19' 2024-03-27T06:17:45.972 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/15' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/20' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/12' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/10' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/11' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/16' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/14' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/18' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/13' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/169/17' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/169' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/19' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/15' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/20' 2024-03-27T06:17:45.973 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/12' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/10' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/11' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/16' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/14' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/18' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/13' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/154/17' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/154' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/19' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/15' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/20' 2024-03-27T06:17:45.974 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/12' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/10' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/11' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/16' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/14' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/18' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/13' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/171/17' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/171' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/19' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/15' 2024-03-27T06:17:45.975 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/20' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/12' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/10' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/11' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/16' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/14' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/18' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/13' 2024-03-27T06:17:46.065 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/139/17' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/139' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/19' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/15' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/20' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/12' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/10' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/11' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/16' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/14' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/18' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/13' 2024-03-27T06:17:46.066 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/102/17' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/102' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/19' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/15' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/20' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/12' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/10' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/11' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/16' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/14' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/18' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/13' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed './copyofsnap1/test1/129/17' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1/129' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1/test1' 2024-03-27T06:17:46.067 INFO:tasks.workunit.client.0.smithi132.stdout:removed directory './copyofsnap1' 2024-03-27T06:17:46.068 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:17:46.069 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:17:46.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:17:46.167 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-03-27T06:17:46.167 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-27T06:17:46.233 INFO:tasks.workunit.client.0.smithi132.stderr:+ set -e 2024-03-27T06:17:46.233 INFO:tasks.workunit.client.0.smithi132.stderr:+ touch foo 2024-03-27T06:17:46.237 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod +x foo 2024-03-27T06:17:46.239 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir .snap/s 2024-03-27T06:17:46.277 INFO:tasks.workunit.client.0.smithi132.stderr:+ find .snap/s/foo -executable 2024-03-27T06:17:46.278 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep foo 2024-03-27T06:17:46.281 INFO:tasks.workunit.client.0.smithi132.stdout:.snap/s/foo 2024-03-27T06:17:46.282 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir .snap/s 2024-03-27T06:17:46.293 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm foo 2024-03-27T06:17:46.300 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:17:46.301 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:17:46.302 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:17:46.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:17:46.388 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-03-27T06:17:46.389 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-27T06:17:46.452 INFO:tasks.workunit.client.0.smithi132.stderr:+ set -e 2024-03-27T06:17:46.453 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir foo 2024-03-27T06:17:46.456 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo x 2024-03-27T06:17:46.457 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:17:46.805 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir foo/.snap/ss 2024-03-27T06:17:46.836 INFO:tasks.workunit.client.0.smithi132.stderr:+ ln foo/x foo/xx 2024-03-27T06:17:46.841 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat foo/.snap/ss/x 2024-03-27T06:17:46.850 INFO:tasks.workunit.client.0.smithi132.stdout:x 2024-03-27T06:17:46.851 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir foo/.snap/ss 2024-03-27T06:17:46.861 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo a 2024-03-27T06:17:46.862 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo b 2024-03-27T06:17:46.863 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir foo/.snap/s 2024-03-27T06:17:46.901 INFO:tasks.workunit.client.0.smithi132.stderr:++ cat foo/.snap/s/a 2024-03-27T06:17:46.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ r=a 2024-03-27T06:17:46.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ '[' -z a ']' 2024-03-27T06:17:46.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ ln foo/b foo/b2 2024-03-27T06:17:46.947 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat foo/.snap/s/b 2024-03-27T06:17:46.956 INFO:tasks.workunit.client.0.smithi132.stdout:b 2024-03-27T06:17:46.957 INFO:tasks.workunit.client.0.smithi132.stdout:this used to hang: 2024-03-27T06:17:46.957 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'this used to hang:' 2024-03-27T06:17:46.957 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo more 2024-03-27T06:17:46.957 INFO:tasks.workunit.client.0.smithi132.stdout:oh, it didn't hang! good job. 2024-03-27T06:17:46.958 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-03-27T06:17:46.958 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat foo/b 2024-03-27T06:17:46.959 INFO:tasks.workunit.client.0.smithi132.stdout:b 2024-03-27T06:17:46.959 INFO:tasks.workunit.client.0.smithi132.stdout:more 2024-03-27T06:17:46.959 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir foo/.snap/s 2024-03-27T06:17:46.970 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm -r foo 2024-03-27T06:17:46.986 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:17:46.987 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:17:46.987 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:17:46.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:17:47.064 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-03-27T06:17:47.065 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-27T06:17:48.596 INFO:tasks.workunit.client.0.smithi132.stderr:+ set -e 2024-03-27T06:17:48.596 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:17:48.600 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir .snap/bar 2024-03-27T06:17:48.637 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/foo 2024-03-27T06:17:48.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/foo 2024-03-27T06:17:48.679 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:17:48.685 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/bar/a 2024-03-27T06:17:48.689 INFO:tasks.workunit.client.0.smithi132.stdout: File: .snap/bar/a 2024-03-27T06:17:48.689 INFO:tasks.workunit.client.0.smithi132.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-27T06:17:48.689 INFO:tasks.workunit.client.0.smithi132.stdout:Device: fch/252d Inode: 1099511633797 Links: 2 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout:Access: 2024-03-27 06:17:48.608905178 +0000 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout:Modify: 2024-03-27 06:17:48.608905178 +0000 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout:Change: 2024-03-27 06:17:48.608905178 +0000 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stdout: Birth: 2024-03-27 06:17:48.608905178 +0000 2024-03-27T06:17:48.690 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir .snap/bar 2024-03-27T06:17:48.702 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:17:48.702 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:17:48.703 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:17:48.704 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:17:48.782 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-03-27T06:17:48.782 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-27T06:17:52.002 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:52.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.234+0000 7f5582c00640 1 Processor -- start 2024-03-27T06:17:52.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.234+0000 7f5582c00640 1 -- start start 2024-03-27T06:17:52.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.235+0000 7f5582c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f557c06b520 0x7f557c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.235+0000 7f5582c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f557c15aa00 0x7f557c15ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.235+0000 7f5582c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c15b790 con 0x7f557c0b11e0 2024-03-27T06:17:52.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.235+0000 7f5582c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f557c0c1080 con 0x7f557c06b520 2024-03-27T06:17:52.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.235+0000 7f5582c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f557c0c1c00 con 0x7f557c15aa00 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f557be00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f557c0b11e0 0x7f557c0c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:43418/0 (socket says 172.21.15.132:43418) 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f557be00640 1 -- 172.21.15.132:0/2369633215 learned_addr learned my addr 172.21.15.132:0/2369633215 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f5580c00640 1 --2- 172.21.15.132:0/2369633215 >> v2:172.21.15.134:3300/0 conn(0x7f557c15aa00 0x7f557c15ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f557b400640 1 --2- 172.21.15.132:0/2369633215 >> v2:172.21.15.132:3300/0 conn(0x7f557c06b520 0x7f557c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f557aa00640 1 -- 172.21.15.132:0/2369633215 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638157068 0 0) 0x7f557c15b790 con 0x7f557c0b11e0 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f5580c00640 1 -- 172.21.15.132:0/2369633215 >> v2:172.21.15.132:3300/0 conn(0x7f557c06b520 msgr2=0x7f557c06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f5580c00640 1 --2- 172.21.15.132:0/2369633215 >> v2:172.21.15.132:3300/0 conn(0x7f557c06b520 0x7f557c06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f5580c00640 1 -- 172.21.15.132:0/2369633215 >> v1:172.21.15.134:6789/0 conn(0x7f557c0b11e0 legacy=0x7f557c0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.236+0000 7f5580c00640 1 -- 172.21.15.132:0/2369633215 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c0c12d0 con 0x7f557c15aa00 2024-03-27T06:17:52.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.237+0000 7f5580c00640 1 --2- 172.21.15.132:0/2369633215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c15ae70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f557c0b1ae0 tx=0x7f557402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95a576bbed98eb63 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.237+0000 7f557aa00640 1 -- 172.21.15.132:0/2369633215 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574039070 con 0x7f557c15aa00 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.237+0000 7f557aa00640 1 -- 172.21.15.132:0/2369633215 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557402cd10 con 0x7f557c15aa00 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.237+0000 7f557aa00640 1 -- 172.21.15.132:0/2369633215 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574034660 con 0x7f557c15aa00 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 -- 172.21.15.132:0/2369633215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 msgr2=0x7f557c15ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 --2- 172.21.15.132:0/2369633215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c15ae70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f557c0b1ae0 tx=0x7f557402bf60 comp rx=0 tx=0).stop 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 -- 172.21.15.132:0/2369633215 shutdown_connections 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 --2- 172.21.15.132:0/2369633215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c15ae70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 --2- 172.21.15.132:0/2369633215 >> v2:172.21.15.132:3300/0 conn(0x7f557c06b520 0x7f557c06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 -- 172.21.15.132:0/2369633215 >> 172.21.15.132:0/2369633215 conn(0x7f557c0cc060 msgr2=0x7f557c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 -- 172.21.15.132:0/2369633215 shutdown_connections 2024-03-27T06:17:52.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.238+0000 7f5582c00640 1 -- 172.21.15.132:0/2369633215 wait complete. 2024-03-27T06:17:52.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.241+0000 7f5582c00640 1 Processor -- start 2024-03-27T06:17:52.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.241+0000 7f5582c00640 1 -- start start 2024-03-27T06:17:52.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06b520 0x7f557c1e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c0b11e0 0x7f557c0c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f557c0b15c0 con 0x7f557c0b11e0 2024-03-27T06:17:52.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f557c0c1890 con 0x7f557c15aa00 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5582c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f557c0c2090 con 0x7f557c06b520 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5580c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f557be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06b520 0x7f557c1e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f557b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c0b11e0 0x7f557c0c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5580c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:38524/0 (socket says 172.21.15.132:38524) 2024-03-27T06:17:52.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.243+0000 7f5580c00640 1 -- 172.21.15.132:0/2894967351 learned_addr learned my addr 172.21.15.132:0/2894967351 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 -- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06b520 msgr2=0x7f557c1e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06b520 0x7f557c1e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 -- 172.21.15.132:0/2894967351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c0b11e0 msgr2=0x7f557c0c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c0b11e0 0x7f557c0c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 -- 172.21.15.132:0/2894967351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c1f0a30 con 0x7f557c15aa00 2024-03-27T06:17:52.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5580c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f557402c570 tx=0x7f5574004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574039050 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5574045070 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557403e830 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.244+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c0c57e0 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.245+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c1f0fd0 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.246+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557c0b2400 con 0x7f557c15aa00 2024-03-27T06:17:52.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.246+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f557403e9d0 con 0x7f557c15aa00 2024-03-27T06:17:52.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.247+0000 7f5573e00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 0x7f555c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.247+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5982+0+0 (secure 0 0 0) 0x7f55740ae610 con 0x7f557c15aa00 2024-03-27T06:17:52.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.248+0000 7f557be00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 0x7f555c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.248+0000 7f557be00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 0x7f555c06e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f556c000f60 tx=0x7f556c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.252+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f557c0b2400 con 0x7f557c15aa00 2024-03-27T06:17:52.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.579+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f557c15a620 con 0x7f557c15aa00 2024-03-27T06:17:52.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.579+0000 7f5573e00640 1 -- 172.21.15.132:0/2894967351 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7f557c15a620 con 0x7f557c15aa00 2024-03-27T06:17:52.570 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:52.570 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:17:52.570 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:17:52.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 msgr2=0x7f555c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 0x7f555c06e9f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f556c000f60 tx=0x7f556c007d50 comp rx=0 tx=0).stop 2024-03-27T06:17:52.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 msgr2=0x7f557c0c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f557402c570 tx=0x7f5574004a40 comp rx=0 tx=0).stop 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 shutdown_connections 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f555c06c530 0x7f555c06e9f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c15aa00 0x7f557c0c3400 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f557c0b11e0 0x7f557c0c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 --2- 172.21.15.132:0/2894967351 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f557c06b520 0x7f557c1e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.582+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 >> 172.21.15.132:0/2894967351 conn(0x7f557c0cc060 msgr2=0x7f557c159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.583+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 shutdown_connections 2024-03-27T06:17:52.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.583+0000 7f5582c00640 1 -- 172.21.15.132:0/2894967351 wait complete. 2024-03-27T06:17:52.593 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2024-03-27T06:17:52.593 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-27T06:17:52.593 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:17:52.602 INFO:tasks.ceph.mds.b:Stopped 2024-03-27T06:17:52.602 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:52.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.799+0000 7f89c5200640 1 Processor -- start 2024-03-27T06:17:52.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.799+0000 7f89c5200640 1 -- start start 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.800+0000 7f89c5200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f89c00c9c80 0x7f89c00ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89c5200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f89c006fb10 0x7f89c00ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89c5200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c0158670 con 0x7f89c0158290 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89c5200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f89c006ed40 con 0x7f89c006fb10 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89c5200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f89c015b2f0 con 0x7f89c00c9c80 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bd400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f89c00c9c80 0x7f89c00ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bde00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f89c0158290 0x7f89c015abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:55716/0 (socket says 172.21.15.132:55716) 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bde00640 1 -- 172.21.15.132:0/3788845383 learned_addr learned my addr 172.21.15.132:0/3788845383 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:17:52.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89be800640 1 --2- 172.21.15.132:0/3788845383 >> v2:172.21.15.132:3300/0 conn(0x7f89c006fb10 0x7f89c00ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bd400640 1 -- 172.21.15.132:0/3788845383 >> v2:172.21.15.132:3300/0 conn(0x7f89c006fb10 msgr2=0x7f89c00ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:52.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bd400640 1 --2- 172.21.15.132:0/3788845383 >> v2:172.21.15.132:3300/0 conn(0x7f89c006fb10 0x7f89c00ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bd400640 1 -- 172.21.15.132:0/3788845383 >> v1:172.21.15.132:6789/0 conn(0x7f89c0158290 legacy=0x7f89c015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.801+0000 7f89bd400640 1 -- 172.21.15.132:0/3788845383 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c006e2f0 con 0x7f89c00c9c80 2024-03-27T06:17:52.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.802+0000 7f89bd400640 1 --2- 172.21.15.132:0/3788845383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00ce290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b36d0 tx=0x7f89ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f63c9d5e3219e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89bca00640 1 -- 172.21.15.132:0/3788845383 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0315a0 con 0x7f89c00c9c80 2024-03-27T06:17:52.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89bca00640 1 -- 172.21.15.132:0/3788845383 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ac037070 con 0x7f89c00c9c80 2024-03-27T06:17:52.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89bca00640 1 -- 172.21.15.132:0/3788845383 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac031960 con 0x7f89c00c9c80 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 -- 172.21.15.132:0/3788845383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 msgr2=0x7f89c00ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 --2- 172.21.15.132:0/3788845383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00ce290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b36d0 tx=0x7f89ac02b840 comp rx=0 tx=0).stop 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 -- 172.21.15.132:0/3788845383 shutdown_connections 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 --2- 172.21.15.132:0/3788845383 >> v2:172.21.15.132:3300/0 conn(0x7f89c006fb10 0x7f89c00ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 --2- 172.21.15.132:0/3788845383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00ce290 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.803+0000 7f89c5200640 1 -- 172.21.15.132:0/3788845383 >> 172.21.15.132:0/3788845383 conn(0x7f89c0067280 msgr2=0x7f89c0067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:52.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.804+0000 7f89c5200640 1 -- 172.21.15.132:0/3788845383 shutdown_connections 2024-03-27T06:17:52.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.804+0000 7f89c5200640 1 -- 172.21.15.132:0/3788845383 wait complete. 2024-03-27T06:17:52.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.806+0000 7f89c5200640 1 Processor -- start 2024-03-27T06:17:52.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.806+0000 7f89c5200640 1 -- start start 2024-03-27T06:17:52.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 0x7f89c0157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 0x7f89c01f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89c006f690 con 0x7f89c006fb10 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f89c006c9b0 con 0x7f89c00c9c80 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89c5200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f89c006e5d0 con 0x7f89c0158290 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89bde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 0x7f89c0157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89bd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89be800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 0x7f89c01f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89bde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 0x7f89c0157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39568/0 (socket says 172.21.15.132:39568) 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.808+0000 7f89bde00640 1 -- 172.21.15.132:0/2891496065 learned_addr learned my addr 172.21.15.132:0/2891496065 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 -- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 msgr2=0x7f89c00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 -- 172.21.15.132:0/2891496065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 msgr2=0x7f89c0157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:52.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 0x7f89c0157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:52.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 -- 172.21.15.132:0/2891496065 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c0150810 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89be800640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 0x7f89c01f0090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f89b800eba0 tx=0x7f89b800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8018020 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b800cd20 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80146c0 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c0150ab0 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.809+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c0158fb0 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.810+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c0068840 con 0x7f89c0158290 2024-03-27T06:17:52.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.811+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f89b8005d10 con 0x7f89c0158290 2024-03-27T06:17:52.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.812+0000 7f89b6000640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:52.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.812+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 5982+0+0 (secure 0 0 0) 0x7f89b808c370 con 0x7f89c0158290 2024-03-27T06:17:52.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.813+0000 7f89bde00640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:52.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.813+0000 7f89bde00640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 0x7f89a006e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89b0004e40 tx=0x7f89b0007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:52.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:52.821+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f89b8022030 con 0x7f89c0158290 2024-03-27T06:17:53.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.060+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c006f8f0 con 0x7f89c0158290 2024-03-27T06:17:53.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.061+0000 7f89b6000640 1 -- 172.21.15.132:0/2891496065 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7f89b8010040 con 0x7f89c0158290 2024-03-27T06:17:53.051 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:53.051 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:17:53.051 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:17:53.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 msgr2=0x7f89a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:53.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 0x7f89a006e9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89b0004e40 tx=0x7f89b0007f10 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 msgr2=0x7f89c01f0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 0x7f89c01f0090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f89b800eba0 tx=0x7f89b800c490 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 shutdown_connections 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89c0158290 0x7f89c01f0090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89a006c530 0x7f89a006e9f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89c00c9c80 0x7f89c00cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 --2- 172.21.15.132:0/2891496065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c006fb10 0x7f89c0157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 >> 172.21.15.132:0/2891496065 conn(0x7f89c0067280 msgr2=0x7f89c00c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 shutdown_connections 2024-03-27T06:17:53.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:53.063+0000 7f89c5200640 1 -- 172.21.15.132:0/2891496065 wait complete. 2024-03-27T06:17:53.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:17:55.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.304+0000 7f74d6400640 1 Processor -- start 2024-03-27T06:17:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.304+0000 7f74d6400640 1 -- start start 2024-03-27T06:17:55.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f74d0152950 0x7f74d01590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f74d0073550 0x7f74d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0073970 0x7f74d00716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f74d0152d30 con 0x7f74d0073550 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f74d006b630 con 0x7f74d0152950 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74d6400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f74d006b8d0 con 0x7f74d0073970 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f74d0073550 0x7f74d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f74d0152950 0x7f74d01590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0073970 0x7f74d00716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f74d0152950 0x7f74d01590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:38556/0 (socket says 172.21.15.132:38556) 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 -- 172.21.15.132:0/3581898357 learned_addr learned my addr 172.21.15.132:0/3581898357 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 -- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0073970 msgr2=0x7f74d00716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 --2- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0073970 0x7f74d00716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 -- 172.21.15.132:0/3581898357 >> v2:172.21.15.132:3300/0 conn(0x7f74d0073550 msgr2=0x7f74d0071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 --2- 172.21.15.132:0/3581898357 >> v2:172.21.15.132:3300/0 conn(0x7f74d0073550 0x7f74d0071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.306+0000 7f74cf400640 1 -- 172.21.15.132:0/3581898357 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d006bbb0 con 0x7f74d0152950 2024-03-27T06:17:55.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74cf400640 1 --2- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0152950 0x7f74d01590a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f74d00c32b0 tx=0x7f74c002b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e74442fac8b9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74ce000640 1 -- 172.21.15.132:0/3581898357 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0035020 con 0x7f74d0152950 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74ce000640 1 -- 172.21.15.132:0/3581898357 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c002d040 con 0x7f74d0152950 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74ce000640 1 -- 172.21.15.132:0/3581898357 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0031a80 con 0x7f74d0152950 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74d6400640 1 -- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0152950 msgr2=0x7f74d01590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.308+0000 7f74d6400640 1 --2- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0152950 0x7f74d01590a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f74d00c32b0 tx=0x7f74c002b900 comp rx=0 tx=0).stop 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 -- 172.21.15.132:0/3581898357 shutdown_connections 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 --2- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0073970 0x7f74d00716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 --2- 172.21.15.132:0/3581898357 >> v2:172.21.15.132:3300/0 conn(0x7f74d0073550 0x7f74d0071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 --2- 172.21.15.132:0/3581898357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0152950 0x7f74d01590a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 -- 172.21.15.132:0/3581898357 >> 172.21.15.132:0/3581898357 conn(0x7f74d00cc060 msgr2=0x7f74d00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 -- 172.21.15.132:0/3581898357 shutdown_connections 2024-03-27T06:17:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.309+0000 7f74d6400640 1 -- 172.21.15.132:0/3581898357 wait complete. 2024-03-27T06:17:55.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.310+0000 7f74d6400640 1 Processor -- start 2024-03-27T06:17:55.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.310+0000 7f74d6400640 1 -- start start 2024-03-27T06:17:55.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d0073550 0x7f74d00afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0152950 0x7f74d01510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74d006c450 con 0x7f74d0073550 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74d006c170 con 0x7f74d0073970 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.311+0000 7f74d6400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f74d006cf50 con 0x7f74d0152950 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cf400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d0073550 0x7f74d00afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:38564/0 (socket says 172.21.15.132:38564) 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0152950 0x7f74d01510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 -- 172.21.15.132:0/3863723830 learned_addr learned my addr 172.21.15.132:0/3863723830 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 -- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0152950 msgr2=0x7f74d01510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0152950 0x7f74d01510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 -- 172.21.15.132:0/3863723830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d0073550 msgr2=0x7f74d00afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d0073550 0x7f74d00afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 -- 172.21.15.132:0/3863723830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d01f0de0 con 0x7f74d0073970 2024-03-27T06:17:55.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74cea00640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00e9b0 tx=0x7f74bc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc017070 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74bc00c9a0 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc00cd90 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.312+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d01f10a0 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.313+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d00c38e0 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.314+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d0068840 con 0x7f74d0073970 2024-03-27T06:17:55.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.315+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f74bc01a070 con 0x7f74d0073970 2024-03-27T06:17:55.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.315+0000 7f74c7400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 0x7f74b006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:55.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.316+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5982+0+0 (secure 0 0 0) 0x7f74bc08b720 con 0x7f74d0073970 2024-03-27T06:17:55.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.316+0000 7f74cf400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 0x7f74b006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:55.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.316+0000 7f74cf400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 0x7f74b006e9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f74c0036000 tx=0x7f74c003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:55.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.320+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f74bc0595e0 con 0x7f74d0073970 2024-03-27T06:17:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.569+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74d0152300 con 0x7f74d0073970 2024-03-27T06:17:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.569+0000 7f74c7400640 1 -- 172.21.15.132:0/3863723830 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7f74bc05d8e0 con 0x7f74d0073970 2024-03-27T06:17:55.560 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:55.560 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:17:55.560 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.572+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 msgr2=0x7f74b006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.572+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 0x7f74b006e9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f74c0036000 tx=0x7f74c003c040 comp rx=0 tx=0).stop 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 msgr2=0x7f74d00c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00e9b0 tx=0x7f74bc00ee70 comp rx=0 tx=0).stop 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 shutdown_connections 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f74d0152950 0x7f74d01510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f74b006c530 0x7f74b006e9f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74d0073970 0x7f74d00c43c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 --2- 172.21.15.132:0/3863723830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d0073550 0x7f74d00afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:55.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 >> 172.21.15.132:0/3863723830 conn(0x7f74d00cc060 msgr2=0x7f74d00ca430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:55.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 shutdown_connections 2024-03-27T06:17:55.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:55.573+0000 7f74d6400640 1 -- 172.21.15.132:0/3863723830 wait complete. 2024-03-27T06:17:55.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:17:57.583 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:17:57.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.803+0000 7fa516800640 1 Processor -- start 2024-03-27T06:17:57.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.803+0000 7fa516800640 1 -- start start 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa516800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 0x7fa51015c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa516800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa5100c3b50 0x7fa51015c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa516800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5100c6760 con 0x7fa5100c7ee0 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa516800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa5100c7050 con 0x7fa5101529d0 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa516800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa5100c7860 con 0x7fa5100c3b50 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa50fe00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa5100c7ee0 0x7fa5100c8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:43426/0 (socket says 172.21.15.132:43426) 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa50fe00640 1 -- 172.21.15.132:0/2968074976 learned_addr learned my addr 172.21.15.132:0/2968074976 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa50f400640 1 --2- 172.21.15.132:0/2968074976 >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 0x7fa51015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa507e00640 1 --2- 172.21.15.132:0/2968074976 >> v2:172.21.15.134:3300/0 conn(0x7fa5100c3b50 0x7fa51015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.805+0000 7fa50ea00640 1 -- 172.21.15.132:0/2968074976 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885178561 0 0) 0x7fa5100c6760 con 0x7fa5100c7ee0 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa50ea00640 1 -- 172.21.15.132:0/2968074976 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4f4003520 con 0x7fa5100c7ee0 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa507e00640 1 -- 172.21.15.132:0/2968074976 >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 msgr2=0x7fa51015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa507e00640 1 --2- 172.21.15.132:0/2968074976 >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 0x7fa51015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa507e00640 1 -- 172.21.15.132:0/2968074976 >> v1:172.21.15.134:6789/0 conn(0x7fa5100c7ee0 legacy=0x7fa5100c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa507e00640 1 -- 172.21.15.132:0/2968074976 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5100c72a0 con 0x7fa5100c3b50 2024-03-27T06:17:57.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.806+0000 7fa50f400640 1 --2- 172.21.15.132:0/2968074976 >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 0x7fa51015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:17:57.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.807+0000 7fa507e00640 1 --2- 172.21.15.132:0/2968074976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c3b50 0x7fa51015c840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5100c6d10 tx=0x7fa4f802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52a787271e9761a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:57.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa50ea00640 1 -- 172.21.15.132:0/2968074976 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8038070 con 0x7fa5100c3b50 2024-03-27T06:17:57.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa50ea00640 1 -- 172.21.15.132:0/2968074976 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f8031710 con 0x7fa5100c3b50 2024-03-27T06:17:57.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa50ea00640 1 -- 172.21.15.132:0/2968074976 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8031ae0 con 0x7fa5100c3b50 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 -- 172.21.15.132:0/2968074976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c3b50 msgr2=0x7fa51015c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 --2- 172.21.15.132:0/2968074976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c3b50 0x7fa51015c840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5100c6d10 tx=0x7fa4f802b7b0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 -- 172.21.15.132:0/2968074976 shutdown_connections 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 --2- 172.21.15.132:0/2968074976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c3b50 0x7fa51015c840 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 --2- 172.21.15.132:0/2968074976 >> v2:172.21.15.132:3300/0 conn(0x7fa5101529d0 0x7fa51015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.808+0000 7fa516800640 1 -- 172.21.15.132:0/2968074976 >> 172.21.15.132:0/2968074976 conn(0x7fa5100cc650 msgr2=0x7fa5100cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.809+0000 7fa516800640 1 -- 172.21.15.132:0/2968074976 shutdown_connections 2024-03-27T06:17:57.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.809+0000 7fa516800640 1 -- 172.21.15.132:0/2968074976 wait complete. 2024-03-27T06:17:57.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.810+0000 7fa516800640 1 Processor -- start 2024-03-27T06:17:57.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.810+0000 7fa516800640 1 -- start start 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5100c3b50 0x7fa510072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 0x7fa5101557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa510152db0 con 0x7fa5100c3b50 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa51015bde0 con 0x7fa5100c7ee0 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa516800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa510067760 con 0x7fa5101529d0 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa507e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.811+0000 7fa50f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5100c3b50 0x7fa510072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:38580/0 (socket says 172.21.15.132:38580) 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 -- 172.21.15.132:0/3209742745 learned_addr learned my addr 172.21.15.132:0/3209742745 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa50fe00640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 0x7fa5101557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 -- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 msgr2=0x7fa5101557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 0x7fa5101557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 -- 172.21.15.132:0/3209742745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5100c3b50 msgr2=0x7fa510072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5100c3b50 0x7fa510072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 -- 172.21.15.132:0/3209742745 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5100cc1e0 con 0x7fa5100c7ee0 2024-03-27T06:17:57.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa50fe00640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 0x7fa5101557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.812+0000 7fa507e00640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009890 tx=0x7fa4f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.813+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8042030 con 0x7fa5100c7ee0 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.813+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f803d670 con 0x7fa5100c7ee0 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.813+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5101f3e00 con 0x7fa5100c7ee0 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.813+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8038040 con 0x7fa5100c7ee0 2024-03-27T06:17:57.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.813+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5101f43c0 con 0x7fa5100c7ee0 2024-03-27T06:17:57.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.814+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa510068840 con 0x7fa5100c7ee0 2024-03-27T06:17:57.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.816+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa4f8045070 con 0x7fa5100c7ee0 2024-03-27T06:17:57.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.817+0000 7fa507400640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 0x7fa4ec06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:17:57.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.817+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5982+0+0 (secure 0 0 0) 0x7fa4f80acfe0 con 0x7fa5100c7ee0 2024-03-27T06:17:57.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.817+0000 7fa50f400640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 0x7fa4ec06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:17:57.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.817+0000 7fa50f400640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 0x7fa4ec06e9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa500009560 tx=0x7fa500009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:17:57.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:57.821+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa5100cc1e0 con 0x7fa5100c7ee0 2024-03-27T06:17:58.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.065+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa51006c7c0 con 0x7fa5100c7ee0 2024-03-27T06:17:58.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.066+0000 7fa507400640 1 -- 172.21.15.132:0/3209742745 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7fa4f807aed0 con 0x7fa5100c7ee0 2024-03-27T06:17:58.057 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:17:58.057 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:17:58.057 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 msgr2=0x7fa4ec06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 0x7fa4ec06e9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa500009560 tx=0x7fa500009190 comp rx=0 tx=0).stop 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 msgr2=0x7fa5101552a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009890 tx=0x7fa4f80046c0 comp rx=0 tx=0).stop 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 shutdown_connections 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa5101529d0 0x7fa5101557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:58.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa4ec06c530 0x7fa4ec06e9f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:58.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5100c7ee0 0x7fa5101552a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:58.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 --2- 172.21.15.132:0/3209742745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5100c3b50 0x7fa510072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:17:58.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.069+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 >> 172.21.15.132:0/3209742745 conn(0x7fa5100cc650 msgr2=0x7fa5100c4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:17:58.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.070+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 shutdown_connections 2024-03-27T06:17:58.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:17:58.070+0000 7fa516800640 1 -- 172.21.15.132:0/3209742745 wait complete. 2024-03-27T06:17:58.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:18:00.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:00.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.304+0000 7f76f8c00640 1 Processor -- start 2024-03-27T06:18:00.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.304+0000 7f76f8c00640 1 -- start start 2024-03-27T06:18:00.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f8c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f76f4154990 0x7f76f406b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:00.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f8c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f406ba60 con 0x7f76f40c7ab0 2024-03-27T06:18:00.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f8c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f406caf0 con 0x7f76f415abb0 2024-03-27T06:18:00.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f8c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f76f40c8c70 con 0x7f76f4154990 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f2800640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f76f415abb0 0x7f76f415afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:43438/0 (socket says 172.21.15.132:43438) 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f1e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f76f4154990 0x7f76f406b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f2800640 1 -- 172.21.15.132:0/1881724784 learned_addr learned my addr 172.21.15.132:0/1881724784 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.306+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3882156239 0 0) 0x7f76f406caf0 con 0x7f76f415abb0 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.307+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76d8003520 con 0x7f76f415abb0 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.307+0000 7f76f1e00640 1 -- 172.21.15.132:0/1881724784 >> v1:172.21.15.134:6789/0 conn(0x7f76f415abb0 legacy=0x7f76f415afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.307+0000 7f76f1e00640 1 -- 172.21.15.132:0/1881724784 >> v1:172.21.15.132:6789/0 conn(0x7f76f40c7ab0 legacy=0x7f76f4072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.307+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336785877 0 0) 0x7f76f406ba60 con 0x7f76f40c7ab0 2024-03-27T06:18:00.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.307+0000 7f76f1e00640 1 -- 172.21.15.132:0/1881724784 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f40c8f50 con 0x7f76f4154990 2024-03-27T06:18:00.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.308+0000 7f76f1e00640 1 --2- 172.21.15.132:0/1881724784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f4154990 0x7f76f406b520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009d90 tx=0x7f76e402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea1e267f1aa8163 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:00.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.308+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4039070 con 0x7f76f4154990 2024-03-27T06:18:00.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.308+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e402cd80 con 0x7f76f4154990 2024-03-27T06:18:00.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.308+0000 7f76f1400640 1 -- 172.21.15.132:0/1881724784 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4034660 con 0x7f76f4154990 2024-03-27T06:18:00.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 -- 172.21.15.132:0/1881724784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f4154990 msgr2=0x7f76f406b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 --2- 172.21.15.132:0/1881724784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f4154990 0x7f76f406b520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009d90 tx=0x7f76e402bfd0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 -- 172.21.15.132:0/1881724784 shutdown_connections 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 --2- 172.21.15.132:0/1881724784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f4154990 0x7f76f406b520 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 -- 172.21.15.132:0/1881724784 >> 172.21.15.132:0/1881724784 conn(0x7f76f406e190 msgr2=0x7f76f406e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 -- 172.21.15.132:0/1881724784 shutdown_connections 2024-03-27T06:18:00.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.309+0000 7f76f8c00640 1 -- 172.21.15.132:0/1881724784 wait complete. 2024-03-27T06:18:00.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.311+0000 7f76f8c00640 1 Processor -- start 2024-03-27T06:18:00.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.311+0000 7f76f8c00640 1 -- start start 2024-03-27T06:18:00.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:00.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 0x7f76f41f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:00.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76f415abb0 0x7f76f41f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:00.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f76f40c97f0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f76f40c9510 con 0x7f76f415abb0 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f8c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f76f4154d70 con 0x7f76f4154990 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76f415abb0 0x7f76f41f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76e9400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 0x7f76f41f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:39620/0 (socket says 172.21.15.132:39620) 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f1e00640 1 -- 172.21.15.132:0/923167442 learned_addr learned my addr 172.21.15.132:0/923167442 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f1e00640 1 -- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 msgr2=0x7f76f41f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.312+0000 7f76f1e00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 0x7f76f41f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f1e00640 1 -- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76f415abb0 msgr2=0x7f76f41f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f1e00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76f415abb0 0x7f76f41f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f1e00640 1 -- 172.21.15.132:0/923167442 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f415b450 con 0x7f76f40c7ab0 2024-03-27T06:18:00.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76e9400640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 0x7f76f41f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f1e00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f76e4037040 tx=0x7f76e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4039050 con 0x7f76f40c7ab0 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e403a040 con 0x7f76f40c7ab0 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f41560e0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4003bd0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.313+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f41566a0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.315+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f4068840 con 0x7f76f40c7ab0 2024-03-27T06:18:00.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.315+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f76e4004030 con 0x7f76f40c7ab0 2024-03-27T06:18:00.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.316+0000 7f76eaa00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 0x7f76d006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:00.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.316+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5982+0+0 (secure 0 0 0) 0x7f76e40ae020 con 0x7f76f40c7ab0 2024-03-27T06:18:00.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.317+0000 7f76e9400640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 0x7f76d006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:00.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.317+0000 7f76e9400640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 0x7f76d006e9f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f76f406c870 tx=0x7f76dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:00.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.321+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f76e407bee0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.618+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76f4153790 con 0x7f76f40c7ab0 2024-03-27T06:18:00.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.619+0000 7f76eaa00640 1 -- 172.21.15.132:0/923167442 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7f76e40801e0 con 0x7f76f40c7ab0 2024-03-27T06:18:00.609 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:00.610 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:18:00.610 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.622+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 msgr2=0x7f76d006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.622+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 0x7f76d006e9f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f76f406c870 tx=0x7f76dc009040 comp rx=0 tx=0).stop 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.622+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 msgr2=0x7f76f41578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.622+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f76e4037040 tx=0x7f76e4004a40 comp rx=0 tx=0).stop 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 shutdown_connections 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76d006c530 0x7f76d006e9f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76f415abb0 0x7f76f41f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f76f4154990 0x7f76f41f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 --2- 172.21.15.132:0/923167442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76f40c7ab0 0x7f76f41578c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:00.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 >> 172.21.15.132:0/923167442 conn(0x7f76f406e190 msgr2=0x7f76f4152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:00.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 shutdown_connections 2024-03-27T06:18:00.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:00.623+0000 7f76f8c00640 1 -- 172.21.15.132:0/923167442 wait complete. 2024-03-27T06:18:00.633 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:18:02.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:02.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.858+0000 7ff92e400640 1 Processor -- start 2024-03-27T06:18:02.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.858+0000 7ff92e400640 1 -- start start 2024-03-27T06:18:02.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.859+0000 7ff92e400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff928154e00 0x7ff92815b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.859+0000 7ff92e400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92806ba60 con 0x7ff92806c3d0 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.859+0000 7ff92e400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92806bd40 con 0x7ff92815bf30 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.859+0000 7ff92e400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff92815ba80 con 0x7ff928154e00 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff926a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff92815bf30 0x7ff9280c8d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53606/0 (socket says 172.21.15.132:53606) 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff927400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff928154e00 0x7ff92815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:02.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff926a00640 1 -- 172.21.15.132:0/276101978 learned_addr learned my addr 172.21.15.132:0/276101978 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:02.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff926000640 1 -- 172.21.15.132:0/276101978 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3195259654 0 0) 0x7ff92806bd40 con 0x7ff92815bf30 2024-03-27T06:18:02.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff926000640 1 -- 172.21.15.132:0/276101978 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff90c003520 con 0x7ff92815bf30 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff927400640 1 -- 172.21.15.132:0/276101978 >> v1:172.21.15.134:6789/0 conn(0x7ff92815bf30 legacy=0x7ff9280c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff927400640 1 -- 172.21.15.132:0/276101978 >> v1:172.21.15.132:6789/0 conn(0x7ff92806c3d0 legacy=0x7ff9280c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.860+0000 7ff927400640 1 -- 172.21.15.132:0/276101978 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9280c8230 con 0x7ff928154e00 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.861+0000 7ff927400640 1 --2- 172.21.15.132:0/276101978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff92815b540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9180097a0 tx=0x7ff91802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f357fe5e6009d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.861+0000 7ff926000640 1 -- 172.21.15.132:0/276101978 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918034070 con 0x7ff928154e00 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.861+0000 7ff926000640 1 -- 172.21.15.132:0/276101978 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff918030610 con 0x7ff928154e00 2024-03-27T06:18:02.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.861+0000 7ff926000640 1 -- 172.21.15.132:0/276101978 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9180309c0 con 0x7ff928154e00 2024-03-27T06:18:02.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 -- 172.21.15.132:0/276101978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 msgr2=0x7ff92815b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:02.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 --2- 172.21.15.132:0/276101978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff92815b540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9180097a0 tx=0x7ff91802b960 comp rx=0 tx=0).stop 2024-03-27T06:18:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 -- 172.21.15.132:0/276101978 shutdown_connections 2024-03-27T06:18:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 --2- 172.21.15.132:0/276101978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff92815b540 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 -- 172.21.15.132:0/276101978 >> 172.21.15.132:0/276101978 conn(0x7ff9280d0150 msgr2=0x7ff9280d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 -- 172.21.15.132:0/276101978 shutdown_connections 2024-03-27T06:18:02.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.862+0000 7ff92e400640 1 -- 172.21.15.132:0/276101978 wait complete. 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.864+0000 7ff92e400640 1 Processor -- start 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.864+0000 7ff92e400640 1 -- start start 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff9280cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 0x7ff9280cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281551e0 con 0x7ff928154e00 2024-03-27T06:18:02.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280c87f0 con 0x7ff92806c3d0 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff92e400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280c7c60 con 0x7ff92815bf30 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff927400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff926a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff9280cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff927400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54806/0 (socket says 172.21.15.132:54806) 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff927400640 1 -- 172.21.15.132:0/1147111321 learned_addr learned my addr 172.21.15.132:0/1147111321 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.865+0000 7ff927e00640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 0x7ff9280cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 -- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 msgr2=0x7ff9280cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 0x7ff9280cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:02.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 -- 172.21.15.132:0/1147111321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 msgr2=0x7ff9280cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff9280cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 -- 172.21.15.132:0/1147111321 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281e4980 con 0x7ff92806c3d0 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927e00640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 0x7ff9280cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff926a00640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff9280cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff927400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9180097a0 tx=0x7ff9180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918034040 con 0x7ff92806c3d0 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff918030410 con 0x7ff92806c3d0 2024-03-27T06:18:02.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918004230 con 0x7ff92806c3d0 2024-03-27T06:18:02.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.866+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281e4c20 con 0x7ff92806c3d0 2024-03-27T06:18:02.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.867+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9280cfab0 con 0x7ff92806c3d0 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.868+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff928068840 con 0x7ff92806c3d0 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.868+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff918037070 con 0x7ff92806c3d0 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.869+0000 7ff91f400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 0x7ff90806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.869+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5982+0+0 (secure 0 0 0) 0x7ff9180ad2d0 con 0x7ff92806c3d0 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.869+0000 7ff926a00640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 0x7ff90806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:02.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.870+0000 7ff926a00640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 0x7ff90806e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9280c9680 tx=0x7ff914007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:02.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:02.873+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff91807b210 con 0x7ff92806c3d0 2024-03-27T06:18:03.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.111+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9280ca3c0 con 0x7ff92806c3d0 2024-03-27T06:18:03.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.112+0000 7ff91f400640 1 -- 172.21.15.132:0/1147111321 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7ff91807f510 con 0x7ff92806c3d0 2024-03-27T06:18:03.106 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:03.106 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.114+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 msgr2=0x7ff90806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.114+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 0x7ff90806e9f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9280c9680 tx=0x7ff914007d30 comp rx=0 tx=0).stop 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.114+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 msgr2=0x7ff9281f10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.114+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9180097a0 tx=0x7ff9180046c0 comp rx=0 tx=0).stop 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 shutdown_connections 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff92815bf30 0x7ff9280cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff928154e00 0x7ff9280cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:03.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff90806c530 0x7ff90806e9f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:03.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 --2- 172.21.15.132:0/1147111321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff92806c3d0 0x7ff9281f10f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:03.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 >> 172.21.15.132:0/1147111321 conn(0x7ff9280d0150 msgr2=0x7ff92815c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:03.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 shutdown_connections 2024-03-27T06:18:03.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:03.115+0000 7ff92e400640 1 -- 172.21.15.132:0/1147111321 wait complete. 2024-03-27T06:18:03.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:18:03.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:18:03.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:18:05.125 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:05.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.354+0000 7fe62a800640 1 Processor -- start 2024-03-27T06:18:05.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.354+0000 7fe62a800640 1 -- start start 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.355+0000 7fe62a800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe62406fb10 0x7fe6241664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.355+0000 7fe62a800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe624158670 con 0x7fe6240c9c80 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe62a800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62415b2f0 con 0x7fe624158290 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe62a800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fe62406ed40 con 0x7fe62406fb10 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe622a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe6240c9c80 0x7fe6240ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:47832/0 (socket says 172.21.15.132:47832) 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe623e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe62406fb10 0x7fe6241664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe622a00640 1 -- 172.21.15.132:0/2026447459 learned_addr learned my addr 172.21.15.132:0/2026447459 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:05.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe623400640 1 --1- 172.21.15.132:0/2026447459 >> v1:172.21.15.134:6789/0 conn(0x7fe624158290 0x7fe62415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53618/0 (socket says 172.21.15.132:53618) 2024-03-27T06:18:05.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe623e00640 1 -- 172.21.15.132:0/2026447459 >> v1:172.21.15.134:6789/0 conn(0x7fe624158290 legacy=0x7fe62415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe623e00640 1 -- 172.21.15.132:0/2026447459 >> v1:172.21.15.132:6789/0 conn(0x7fe6240c9c80 legacy=0x7fe6240ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe623e00640 1 -- 172.21.15.132:0/2026447459 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62406e2f0 con 0x7fe62406fb10 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.356+0000 7fe622000640 1 -- 172.21.15.132:0/2026447459 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121851960 0 0) 0x7fe62415b2f0 con 0x7fe624158290 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.357+0000 7fe623e00640 1 --2- 172.21.15.132:0/2026447459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe62406fb10 0x7fe6241664c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7121ed3490322af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe622000640 1 -- 172.21.15.132:0/2026447459 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c039070 con 0x7fe62406fb10 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe622000640 1 -- 172.21.15.132:0/2026447459 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe61c02cc60 con 0x7fe62406fb10 2024-03-27T06:18:05.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe622000640 1 -- 172.21.15.132:0/2026447459 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c034660 con 0x7fe62406fb10 2024-03-27T06:18:05.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 -- 172.21.15.132:0/2026447459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe62406fb10 msgr2=0x7fe6241664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 --2- 172.21.15.132:0/2026447459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe62406fb10 0x7fe6241664c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c02beb0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 -- 172.21.15.132:0/2026447459 shutdown_connections 2024-03-27T06:18:05.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 --2- 172.21.15.132:0/2026447459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe62406fb10 0x7fe6241664c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 -- 172.21.15.132:0/2026447459 >> 172.21.15.132:0/2026447459 conn(0x7fe624067280 msgr2=0x7fe624067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:05.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 -- 172.21.15.132:0/2026447459 shutdown_connections 2024-03-27T06:18:05.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.358+0000 7fe62a800640 1 -- 172.21.15.132:0/2026447459 wait complete. 2024-03-27T06:18:05.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.360+0000 7fe62a800640 1 Processor -- start 2024-03-27T06:18:05.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.360+0000 7fe62a800640 1 -- start start 2024-03-27T06:18:05.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 0x7fe624155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 0x7fe6240cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe62406f690 con 0x7fe62406fb10 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe62406c9b0 con 0x7fe6240c9c80 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe62a800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe62406e5d0 con 0x7fe624158290 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe623400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 0x7fe624155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe623e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe622a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 0x7fe6240cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe623e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:58032/0 (socket says 172.21.15.132:58032) 2024-03-27T06:18:05.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe623400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 0x7fe624155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:60244/0 (socket says 172.21.15.132:60244) 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.361+0000 7fe623e00640 1 -- 172.21.15.132:0/1316296160 learned_addr learned my addr 172.21.15.132:0/1316296160 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 -- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 msgr2=0x7fe6240cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 0x7fe6240cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 -- 172.21.15.132:0/1316296160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 msgr2=0x7fe624155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 0x7fe624155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 -- 172.21.15.132:0/1316296160 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6240c3fc0 con 0x7fe624158290 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe622a00640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 0x7fe6240cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe623e00640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c039050 con 0x7fe624158290 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe61c03a040 con 0x7fe624158290 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c03e610 con 0x7fe624158290 2024-03-27T06:18:05.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.362+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6241f4010 con 0x7fe624158290 2024-03-27T06:18:05.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.363+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe624158c20 con 0x7fe624158290 2024-03-27T06:18:05.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.364+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624068840 con 0x7fe624158290 2024-03-27T06:18:05.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.364+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe61c047020 con 0x7fe624158290 2024-03-27T06:18:05.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.365+0000 7fe61b400640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 0x7fe60406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:05.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.365+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 5954+0+0 (secure 0 0 0) 0x7fe61c0adba0 con 0x7fe624158290 2024-03-27T06:18:05.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.365+0000 7fe623400640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 0x7fe60406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:05.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.366+0000 7fe623400640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 0x7fe60406e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe614007f70 tx=0x7fe614007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:05.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.370+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe61c04e020 con 0x7fe624158290 2024-03-27T06:18:05.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.716+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe62406d750 con 0x7fe624158290 2024-03-27T06:18:05.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.717+0000 7fe61b400640 1 -- 172.21.15.132:0/1316296160 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7fe61c07ba00 con 0x7fe624158290 2024-03-27T06:18:05.707 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:05.708 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:18:05.708 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 msgr2=0x7fe60406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 0x7fe60406e9f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe614007f70 tx=0x7fe614007f00 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 msgr2=0x7fe6240ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c004a40 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 shutdown_connections 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe624158290 0x7fe6240ccff0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe60406c530 0x7fe60406e9f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6240c9c80 0x7fe6240cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 --2- 172.21.15.132:0/1316296160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe62406fb10 0x7fe624155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 >> 172.21.15.132:0/1316296160 conn(0x7fe624067280 msgr2=0x7fe6240c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:05.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.720+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 shutdown_connections 2024-03-27T06:18:05.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:05.721+0000 7fe62a800640 1 -- 172.21.15.132:0/1316296160 wait complete. 2024-03-27T06:18:05.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:18:07.730 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:07.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.963+0000 7ff248e00640 1 Processor -- start 2024-03-27T06:18:07.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.963+0000 7ff248e00640 1 -- start start 2024-03-27T06:18:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff248e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff24406b790 0x7ff244152490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff248e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 0x7ff2441529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff248e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24406c170 con 0x7ff244070430 2024-03-27T06:18:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff248e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff24406c450 con 0x7ff24406bb70 2024-03-27T06:18:07.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff248e00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7ff2440c9460 con 0x7ff24406b790 2024-03-27T06:18:07.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff243400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff24406b790 0x7ff244152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff23aa00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 0x7ff2441529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff243e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff244070430 0x7ff2441586f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53626/0 (socket says 172.21.15.132:53626) 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.965+0000 7ff243e00640 1 -- 172.21.15.132:0/1221935795 learned_addr learned my addr 172.21.15.132:0/1221935795 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff242a00640 1 -- 172.21.15.132:0/1221935795 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703486795 0 0) 0x7ff24406c170 con 0x7ff244070430 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff242a00640 1 -- 172.21.15.132:0/1221935795 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff228003520 con 0x7ff244070430 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff243400640 1 -- 172.21.15.132:0/1221935795 >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 msgr2=0x7ff2441529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff243400640 1 --2- 172.21.15.132:0/1221935795 >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 0x7ff2441529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff243400640 1 -- 172.21.15.132:0/1221935795 >> v1:172.21.15.134:6789/0 conn(0x7ff244070430 legacy=0x7ff2441586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff243400640 1 -- 172.21.15.132:0/1221935795 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24415c7a0 con 0x7ff24406b790 2024-03-27T06:18:07.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.966+0000 7ff23aa00640 1 --2- 172.21.15.132:0/1221935795 >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 0x7ff2441529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:07.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff243400640 1 --2- 172.21.15.132:0/1221935795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244152490 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff244071b50 tx=0x7ff23402be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9adda6d230afe66 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:07.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff242a00640 1 -- 172.21.15.132:0/1221935795 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234039070 con 0x7ff24406b790 2024-03-27T06:18:07.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff242a00640 1 -- 172.21.15.132:0/1221935795 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23402cbe0 con 0x7ff24406b790 2024-03-27T06:18:07.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff242a00640 1 -- 172.21.15.132:0/1221935795 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234034440 con 0x7ff24406b790 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff248e00640 1 -- 172.21.15.132:0/1221935795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 msgr2=0x7ff244152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.968+0000 7ff248e00640 1 --2- 172.21.15.132:0/1221935795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244152490 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff244071b50 tx=0x7ff23402be30 comp rx=0 tx=0).stop 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 -- 172.21.15.132:0/1221935795 shutdown_connections 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 --2- 172.21.15.132:0/1221935795 >> v2:172.21.15.132:3300/0 conn(0x7ff24406bb70 0x7ff2441529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 --2- 172.21.15.132:0/1221935795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244152490 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 -- 172.21.15.132:0/1221935795 >> 172.21.15.132:0/1221935795 conn(0x7ff24406f910 msgr2=0x7ff24406fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 -- 172.21.15.132:0/1221935795 shutdown_connections 2024-03-27T06:18:07.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.969+0000 7ff248e00640 1 -- 172.21.15.132:0/1221935795 wait complete. 2024-03-27T06:18:07.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.970+0000 7ff248e00640 1 Processor -- start 2024-03-27T06:18:07.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 -- start start 2024-03-27T06:18:07.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 0x7ff2440c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff244071f20 con 0x7ff24406bb70 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff2440c90f0 con 0x7ff24406b790 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.971+0000 7ff248e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff2440720a0 con 0x7ff244070430 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff23aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 0x7ff2440c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54842/0 (socket says 172.21.15.132:54842) 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff23aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:60272/0 (socket says 172.21.15.132:60272) 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 -- 172.21.15.132:0/3326424561 learned_addr learned my addr 172.21.15.132:0/3326424561 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 -- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 msgr2=0x7ff2440c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 0x7ff2440c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 -- 172.21.15.132:0/3326424561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 msgr2=0x7ff24415b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:07.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 -- 172.21.15.132:0/3326424561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2441590f0 con 0x7ff24406b790 2024-03-27T06:18:07.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 0x7ff2440c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:07.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff23aa00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:07.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.972+0000 7ff243400640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff234009e80 tx=0x7ff234004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.973+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234044030 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.973+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23402c9e0 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.973+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2440c4d60 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.973+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234039040 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.973+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2440c5320 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.974+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff244150560 con 0x7ff24406b790 2024-03-27T06:18:07.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.975+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff234047070 con 0x7ff24406b790 2024-03-27T06:18:07.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.977+0000 7ff23be00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 0x7ff22006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:07.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.977+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5954+0+0 (secure 0 0 0) 0x7ff2340add10 con 0x7ff24406b790 2024-03-27T06:18:07.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.977+0000 7ff23aa00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 0x7ff22006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:07.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.978+0000 7ff23aa00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 0x7ff22006ea40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2440c7f40 tx=0x7ff22c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:07.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:07.981+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff244150560 con 0x7ff24406b790 2024-03-27T06:18:08.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.343+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2440686c0 con 0x7ff24406b790 2024-03-27T06:18:08.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.344+0000 7ff23be00640 1 -- 172.21.15.132:0/3326424561 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6886 (secure 0 0 0) 0x7ff2340b3020 con 0x7ff24406b790 2024-03-27T06:18:08.335 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:08.335 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4817,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:16:59.525446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4270,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6838/2702381292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2702381292},{"type":"v1","addr":"172.21.15.134:6838","nonce":2702381292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273,4270]},"id":1}]} 2024-03-27T06:18:08.335 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 24 2024-03-27T06:18:08.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 msgr2=0x7ff22006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 0x7ff22006ea40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2440c7f40 tx=0x7ff22c009190 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 msgr2=0x7ff244158850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff234009e80 tx=0x7ff234004a40 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 shutdown_connections 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff244070430 0x7ff2440c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff24406bb70 0x7ff24415b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff22006c580 0x7ff22006ea40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 --2- 172.21.15.132:0/3326424561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24406b790 0x7ff244158850 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.347+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 >> 172.21.15.132:0/3326424561 conn(0x7ff24406f910 msgr2=0x7ff24406c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.348+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 shutdown_connections 2024-03-27T06:18:08.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:08.348+0000 7ff248e00640 1 -- 172.21.15.132:0/3326424561 wait complete. 2024-03-27T06:18:08.356 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-27T06:18:10.357 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:10.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.578+0000 7f0097400640 1 Processor -- start 2024-03-27T06:18:10.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.578+0000 7f0097400640 1 -- start start 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.580+0000 7f0097400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f009006c8a0 0x7f00900c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.580+0000 7f0097400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f009006cc80 0x7f00900c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.580+0000 7f0097400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00900cdad0 con 0x7f00900c77f0 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.580+0000 7f0097400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f00900cc210 con 0x7f009006c8a0 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.580+0000 7f0097400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f009015b690 con 0x7f009006cc80 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008fe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f009006cc80 0x7f00900c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f0095400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f00900c77f0 0x7f0090154890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53638/0 (socket says 172.21.15.132:53638) 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f0095400640 1 -- 172.21.15.132:0/700938008 learned_addr learned my addr 172.21.15.132:0/700938008 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f0094a00640 1 --2- 172.21.15.132:0/700938008 >> v2:172.21.15.132:3300/0 conn(0x7f009006c8a0 0x7f00900c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008fe00640 1 -- 172.21.15.132:0/700938008 >> v2:172.21.15.132:3300/0 conn(0x7f009006c8a0 msgr2=0x7f00900c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008fe00640 1 --2- 172.21.15.132:0/700938008 >> v2:172.21.15.132:3300/0 conn(0x7f009006c8a0 0x7f00900c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008fe00640 1 -- 172.21.15.132:0/700938008 >> v1:172.21.15.134:6789/0 conn(0x7f00900c77f0 legacy=0x7f0090154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008fe00640 1 -- 172.21.15.132:0/700938008 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009015b970 con 0x7f009006cc80 2024-03-27T06:18:10.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.581+0000 7f008f400640 1 -- 172.21.15.132:0/700938008 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3046033243 0 0) 0x7f00900cdad0 con 0x7f00900c77f0 2024-03-27T06:18:10.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.582+0000 7f008fe00640 1 --2- 172.21.15.132:0/700938008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00900c8860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f00901587a0 tx=0x7f007c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde795b6e691c27a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:10.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.582+0000 7f008f400640 1 -- 172.21.15.132:0/700938008 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c0315a0 con 0x7f009006cc80 2024-03-27T06:18:10.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.582+0000 7f008f400640 1 -- 172.21.15.132:0/700938008 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f007c037070 con 0x7f009006cc80 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.582+0000 7f008f400640 1 -- 172.21.15.132:0/700938008 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c031960 con 0x7f009006cc80 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 -- 172.21.15.132:0/700938008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 msgr2=0x7f00900c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 --2- 172.21.15.132:0/700938008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00900c8860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f00901587a0 tx=0x7f007c02b840 comp rx=0 tx=0).stop 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 -- 172.21.15.132:0/700938008 shutdown_connections 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 --2- 172.21.15.132:0/700938008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00900c8860 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 --2- 172.21.15.132:0/700938008 >> v2:172.21.15.132:3300/0 conn(0x7f009006c8a0 0x7f00900c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 -- 172.21.15.132:0/700938008 >> 172.21.15.132:0/700938008 conn(0x7f00900ccbe0 msgr2=0x7f00900ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 -- 172.21.15.132:0/700938008 shutdown_connections 2024-03-27T06:18:10.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.583+0000 7f0097400640 1 -- 172.21.15.132:0/700938008 wait complete. 2024-03-27T06:18:10.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.585+0000 7f0097400640 1 Processor -- start 2024-03-27T06:18:10.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.585+0000 7f0097400640 1 -- start start 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00901f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00900c77f0 0x7f00901f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f009006d440 con 0x7f00900c77f0 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f009006d2c0 con 0x7f009006cc80 2024-03-27T06:18:10.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0097400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f00900cd700 con 0x7f009006c8a0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f008fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00901f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0094a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0094a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:58046/0 (socket says 172.21.15.132:58046) 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0094a00640 1 -- 172.21.15.132:0/1851934360 learned_addr learned my addr 172.21.15.132:0/1851934360 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0095400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00900c77f0 0x7f00901f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0094a00640 1 -- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 msgr2=0x7f00901f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.586+0000 7f0094a00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00901f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0094a00640 1 -- 172.21.15.132:0/1851934360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00900c77f0 msgr2=0x7f00901f0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0094a00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00900c77f0 0x7f00901f0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0094a00640 1 -- 172.21.15.132:0/1851934360 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00900cfbd0 con 0x7f009006c8a0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0094a00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f008400e8a0 tx=0x7f008400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:10.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084017070 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f008400c940 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0090149310 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008400cd30 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.587+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0090149580 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.589+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0090158df0 con 0x7f009006c8a0 2024-03-27T06:18:10.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.589+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f008401a070 con 0x7f009006c8a0 2024-03-27T06:18:10.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.590+0000 7f008cc00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 0x7f007006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:10.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.590+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f008408b6c0 con 0x7f009006c8a0 2024-03-27T06:18:10.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.590+0000 7f008fe00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 0x7f007006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:10.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.591+0000 7f008fe00640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 0x7f007006e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f007c036000 tx=0x7f007c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:10.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.599+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f0090158df0 con 0x7f009006c8a0 2024-03-27T06:18:10.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.837+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00900c99d0 con 0x7f009006c8a0 2024-03-27T06:18:10.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.838+0000 7f008cc00640 1 -- 172.21.15.132:0/1851934360 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6054 (secure 0 0 0) 0x7f00900c99d0 con 0x7f009006c8a0 2024-03-27T06:18:10.828 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:10.828 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:09.157697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718]},"id":1}]} 2024-03-27T06:18:10.829 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 25 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.840+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 msgr2=0x7f007006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 0x7f007006e9f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f007c036000 tx=0x7f007c03d040 comp rx=0 tx=0).stop 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 msgr2=0x7f00900c2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f008400e8a0 tx=0x7f008400ed60 comp rx=0 tx=0).stop 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 shutdown_connections 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00900c77f0 0x7f00901f0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f007006c530 0x7f007006e9f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f009006cc80 0x7f00901f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 --2- 172.21.15.132:0/1851934360 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f009006c8a0 0x7f00900c2090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:10.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 >> 172.21.15.132:0/1851934360 conn(0x7f00900ccbe0 msgr2=0x7f00900ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:10.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 shutdown_connections 2024-03-27T06:18:10.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:10.841+0000 7f0097400640 1 -- 172.21.15.132:0/1851934360 wait complete. 2024-03-27T06:18:10.848 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-03-27T06:18:10.848 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-27T06:18:10.848 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:11.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.059+0000 7fbd12800640 1 Processor -- start 2024-03-27T06:18:11.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.059+0000 7fbd12800640 1 -- start start 2024-03-27T06:18:11.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 0x7fbd0c0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbd0c073880 0x7fbd0c157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fbd0c072be0 con 0x7fbd0c151ad0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fbd0c06b520 con 0x7fbd0c073880 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.060+0000 7fbd12800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0c06b7c0 con 0x7fbd0c1516f0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:58058/0 (socket says 172.21.15.132:58058) 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 -- 172.21.15.132:0/2905746598 learned_addr learned my addr 172.21.15.132:0/2905746598 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd03e00640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 0x7fbd0c0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0be00640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.134:3300/0 conn(0x7fbd0c073880 0x7fbd0c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 -- 172.21.15.132:0/2905746598 >> v2:172.21.15.134:3300/0 conn(0x7fbd0c073880 msgr2=0x7fbd0c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.134:3300/0 conn(0x7fbd0c073880 0x7fbd0c157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 -- 172.21.15.132:0/2905746598 >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 msgr2=0x7fbd0c0c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 0x7fbd0c0c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd0b400640 1 -- 172.21.15.132:0/2905746598 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c06baa0 con 0x7fbd0c1516f0 2024-03-27T06:18:11.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.061+0000 7fbd03e00640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 0x7fbd0c0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.062+0000 7fbd0b400640 1 --2- 172.21.15.132:0/2905746598 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0b1ae0 tx=0x7fbcfc02b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d472a73b94bb341b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.062+0000 7fbd0aa00640 1 -- 172.21.15.132:0/2905746598 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc034070 con 0x7fbd0c1516f0 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.062+0000 7fbd0aa00640 1 -- 172.21.15.132:0/2905746598 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc030860 con 0x7fbd0c1516f0 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.062+0000 7fbd0aa00640 1 -- 172.21.15.132:0/2905746598 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc030c30 con 0x7fbd0c1516f0 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 -- 172.21.15.132:0/2905746598 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 msgr2=0x7fbd0c0c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 --2- 172.21.15.132:0/2905746598 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0b1ae0 tx=0x7fbcfc02b9a0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 -- 172.21.15.132:0/2905746598 shutdown_connections 2024-03-27T06:18:11.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.134:3300/0 conn(0x7fbd0c073880 0x7fbd0c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 --2- 172.21.15.132:0/2905746598 >> v2:172.21.15.132:3300/0 conn(0x7fbd0c151ad0 0x7fbd0c0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 --2- 172.21.15.132:0/2905746598 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c0c7e20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 -- 172.21.15.132:0/2905746598 >> 172.21.15.132:0/2905746598 conn(0x7fbd0c0cc060 msgr2=0x7fbd0c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:11.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 -- 172.21.15.132:0/2905746598 shutdown_connections 2024-03-27T06:18:11.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.063+0000 7fbd12800640 1 -- 172.21.15.132:0/2905746598 wait complete. 2024-03-27T06:18:11.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.065+0000 7fbd12800640 1 Processor -- start 2024-03-27T06:18:11.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.065+0000 7fbd12800640 1 -- start start 2024-03-27T06:18:11.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd0c073880 0x7fbd0c14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 0x7fbd0c0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c06bd80 con 0x7fbd0c073880 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c072870 con 0x7fbd0c151ad0 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd12800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0c0c1080 con 0x7fbd0c1516f0 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd0be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 0x7fbd0c0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd0b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd0c073880 0x7fbd0c14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:58064/0 (socket says 172.21.15.132:58064) 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd0be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 0x7fbd0c0cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54880/0 (socket says 172.21.15.132:54880) 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 -- 172.21.15.132:0/81534209 learned_addr learned my addr 172.21.15.132:0/81534209 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 -- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 msgr2=0x7fbd0c0cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 0x7fbd0c0cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 -- 172.21.15.132:0/81534209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd0c073880 msgr2=0x7fbd0c14c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:11.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd0c073880 0x7fbd0c14c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.066+0000 7fbd03e00640 1 -- 172.21.15.132:0/81534209 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c0c1af0 con 0x7fbd0c1516f0 2024-03-27T06:18:11.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd03e00640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e870 tx=0x7fbcf400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf400f040 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf4004230 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf400c6d0 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0c1f0150 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.067+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0c14ba30 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.068+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0c068840 con 0x7fbd0c1516f0 2024-03-27T06:18:11.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.069+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbcf400c870 con 0x7fbd0c1516f0 2024-03-27T06:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.070+0000 7fbd03400640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 0x7fbce806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.070+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7fbcf408b400 con 0x7fbd0c1516f0 2024-03-27T06:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.070+0000 7fbd0b400640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 0x7fbce806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:11.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.070+0000 7fbd0b400640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 0x7fbce806ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc036000 tx=0x7fbcfc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:11.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.078+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fbcf4017020 con 0x7fbd0c1516f0 2024-03-27T06:18:11.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.326+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd0c0c76f0 con 0x7fbd0c1516f0 2024-03-27T06:18:11.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.327+0000 7fbd03400640 1 -- 172.21.15.132:0/81534209 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6054 (secure 0 0 0) 0x7fbcf4059210 con 0x7fbd0c1516f0 2024-03-27T06:18:11.317 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:11.317 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:09.157697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718]},"id":1}]} 2024-03-27T06:18:11.317 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 25 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 msgr2=0x7fbce806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 0x7fbce806ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc036000 tx=0x7fbcfc03b040 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 msgr2=0x7fbd0c14d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e870 tx=0x7fbcf400ed30 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 shutdown_connections 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbce806c560 0x7fbce806ea20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd0c151ad0 0x7fbd0c0cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd0c1516f0 0x7fbd0c14d100 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.329+0000 7fbd12800640 1 --2- 172.21.15.132:0/81534209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd0c073880 0x7fbd0c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.330+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 >> 172.21.15.132:0/81534209 conn(0x7fbd0c0cc060 msgr2=0x7fbd0c0c6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.330+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 shutdown_connections 2024-03-27T06:18:11.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:11.330+0000 7fbd12800640 1 -- 172.21.15.132:0/81534209 wait complete. 2024-03-27T06:18:11.341 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 21 }, "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:18:09.157697+0000", "qdb_cluster": [ 4273, 4718 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4817, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:18:13.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:13.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.585+0000 7fdc5d600640 1 Processor -- start 2024-03-27T06:18:13.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.585+0000 7fdc5d600640 1 -- start start 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc5d600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdc58151ad0 0x7fdc58157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc5d600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc5806b520 con 0x7fdc58073880 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc5d600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc5806b7c0 con 0x7fdc581516f0 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc5d600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fdc58072be0 con 0x7fdc58151ad0 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc56a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdc581516f0 0x7fdc580c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34802/0 (socket says 172.21.15.132:34802) 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc56a00640 1 -- 172.21.15.132:0/1145433474 learned_addr learned my addr 172.21.15.132:0/1145433474 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:13.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.587+0000 7fdc4e000640 1 --2- 172.21.15.132:0/1145433474 >> v2:172.21.15.132:3300/0 conn(0x7fdc58151ad0 0x7fdc58157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:13.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059173563 0 0) 0x7fdc5806b7c0 con 0x7fdc581516f0 2024-03-27T06:18:13.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc3c003520 con 0x7fdc581516f0 2024-03-27T06:18:13.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc4e000640 1 -- 172.21.15.132:0/1145433474 >> v1:172.21.15.134:6789/0 conn(0x7fdc581516f0 legacy=0x7fdc580c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc4e000640 1 -- 172.21.15.132:0/1145433474 >> v1:172.21.15.132:6789/0 conn(0x7fdc58073880 legacy=0x7fdc58161e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725777071 0 0) 0x7fdc5806b520 con 0x7fdc58073880 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.588+0000 7fdc4e000640 1 -- 172.21.15.132:0/1145433474 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5806baa0 con 0x7fdc58151ad0 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.589+0000 7fdc4e000640 1 --2- 172.21.15.132:0/1145433474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58151ad0 0x7fdc58157570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc400097a0 tx=0x7fdc4002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54430ef82c0c98b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.589+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4002f810 con 0x7fdc58151ad0 2024-03-27T06:18:13.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.589+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc4002f9b0 con 0x7fdc58151ad0 2024-03-27T06:18:13.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.589+0000 7fdc56000640 1 -- 172.21.15.132:0/1145433474 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4002fd80 con 0x7fdc58151ad0 2024-03-27T06:18:13.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 -- 172.21.15.132:0/1145433474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58151ad0 msgr2=0x7fdc58157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 --2- 172.21.15.132:0/1145433474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58151ad0 0x7fdc58157570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc400097a0 tx=0x7fdc4002b9f0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 -- 172.21.15.132:0/1145433474 shutdown_connections 2024-03-27T06:18:13.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 --2- 172.21.15.132:0/1145433474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58151ad0 0x7fdc58157570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 -- 172.21.15.132:0/1145433474 >> 172.21.15.132:0/1145433474 conn(0x7fdc580cc060 msgr2=0x7fdc580cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:13.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 -- 172.21.15.132:0/1145433474 shutdown_connections 2024-03-27T06:18:13.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.590+0000 7fdc5d600640 1 -- 172.21.15.132:0/1145433474 wait complete. 2024-03-27T06:18:13.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.592+0000 7fdc5d600640 1 Processor -- start 2024-03-27T06:18:13.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.592+0000 7fdc5d600640 1 -- start start 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 0x7fdc581570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc58151ad0 0x7fdc581575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5806bd80 con 0x7fdc58073880 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdc58072870 con 0x7fdc58151ad0 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc5d600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdc580c1080 con 0x7fdc581516f0 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc56a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc56a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38502/0 (socket says 172.21.15.132:38502) 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc56a00640 1 -- 172.21.15.132:0/2975983566 learned_addr learned my addr 172.21.15.132:0/2975983566 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc4e000640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 0x7fdc581570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.593+0000 7fdc57400640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc58151ad0 0x7fdc581575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:13.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 -- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc58151ad0 msgr2=0x7fdc581575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc58151ad0 0x7fdc581575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 -- 172.21.15.132:0/2975983566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 msgr2=0x7fdc58141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 -- 172.21.15.132:0/2975983566 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc581f04b0 con 0x7fdc581516f0 2024-03-27T06:18:13.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc56a00640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4e000640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 0x7fdc581570b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdc40033000 tx=0x7fdc40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40004220 con 0x7fdc581516f0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc400043c0 con 0x7fdc581516f0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40037910 con 0x7fdc581516f0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.594+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc581580c0 con 0x7fdc581516f0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.595+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc58154870 con 0x7fdc581516f0 2024-03-27T06:18:13.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.596+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc58068840 con 0x7fdc581516f0 2024-03-27T06:18:13.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.597+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdc40037ab0 con 0x7fdc581516f0 2024-03-27T06:18:13.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.597+0000 7fdc4f400640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 0x7fdc3406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:13.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.598+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7fdc40047070 con 0x7fdc581516f0 2024-03-27T06:18:13.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.598+0000 7fdc56a00640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 0x7fdc3406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:13.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.598+0000 7fdc56a00640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 0x7fdc3406ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc480011e0 tx=0x7fdc48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:13.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.602+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fdc40033030 con 0x7fdc581516f0 2024-03-27T06:18:13.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.908+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc580c7c10 con 0x7fdc581516f0 2024-03-27T06:18:13.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.909+0000 7fdc4f400640 1 -- 172.21.15.132:0/2975983566 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6054 (secure 0 0 0) 0x7fdc4007aee0 con 0x7fdc581516f0 2024-03-27T06:18:13.899 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:13.899 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:09.157697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718]},"id":1}]} 2024-03-27T06:18:13.899 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 25 2024-03-27T06:18:13.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 msgr2=0x7fdc3406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 0x7fdc3406ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc480011e0 tx=0x7fdc48009040 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 msgr2=0x7fdc581570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 0x7fdc581570b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdc40033000 tx=0x7fdc40004730 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 shutdown_connections 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdc3406c560 0x7fdc3406ea20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc58151ad0 0x7fdc581575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdc581516f0 0x7fdc581570b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 --2- 172.21.15.132:0/2975983566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc58073880 0x7fdc58141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 >> 172.21.15.132:0/2975983566 conn(0x7fdc580cc060 msgr2=0x7fdc5806cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.911+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 shutdown_connections 2024-03-27T06:18:13.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:13.912+0000 7fdc5d600640 1 -- 172.21.15.132:0/2975983566 wait complete. 2024-03-27T06:18:13.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:15.922 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:16.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.147+0000 7f07abe00640 1 Processor -- start 2024-03-27T06:18:16.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.147+0000 7f07abe00640 1 -- start start 2024-03-27T06:18:16.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac1547a0 0x7f07ac06b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f07ac073900 0x7f07ac0c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 0x7f07ac0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f07ac15b020 con 0x7f07ac0c7ce0 2024-03-27T06:18:16.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f07ac0c71f0 con 0x7f07ac073900 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.148+0000 7f07abe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f07ac0c74d0 con 0x7f07ac1547a0 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f07ac073900 0x7f07ac0c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07ab400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 0x7f07ac0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f07ac073900 0x7f07ac0c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:35610/0 (socket says 172.21.15.132:35610) 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07ab400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 0x7f07ac0c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38504/0 (socket says 172.21.15.132:38504) 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 -- 172.21.15.132:0/3777538344 learned_addr learned my addr 172.21.15.132:0/3777538344 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aaa00640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac1547a0 0x7f07ac06b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 -- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac1547a0 msgr2=0x7f07ac06b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac1547a0 0x7f07ac06b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 -- 172.21.15.132:0/3777538344 >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 msgr2=0x7f07ac0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 --2- 172.21.15.132:0/3777538344 >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 0x7f07ac0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.149+0000 7f07aa000640 1 -- 172.21.15.132:0/3777538344 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac15a6f0 con 0x7f07ac073900 2024-03-27T06:18:16.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.150+0000 7f07aa000640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac073900 0x7f07ac0c83a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07ac070170 tx=0x7f079802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80746af6b4655e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07a9600640 1 -- 172.21.15.132:0/3777538344 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798034070 con 0x7f07ac073900 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07a9600640 1 -- 172.21.15.132:0/3777538344 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0798030860 con 0x7f07ac073900 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07a9600640 1 -- 172.21.15.132:0/3777538344 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798030c30 con 0x7f07ac073900 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 -- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac073900 msgr2=0x7f07ac0c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac073900 0x7f07ac0c83a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07ac070170 tx=0x7f079802b9a0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 -- 172.21.15.132:0/3777538344 shutdown_connections 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 --2- 172.21.15.132:0/3777538344 >> v2:172.21.15.132:3300/0 conn(0x7f07ac0c7ce0 0x7f07ac0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac073900 0x7f07ac0c83a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 --2- 172.21.15.132:0/3777538344 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac1547a0 0x7f07ac06b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 -- 172.21.15.132:0/3777538344 >> 172.21.15.132:0/3777538344 conn(0x7f07ac0c9000 msgr2=0x7f07ac0c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 -- 172.21.15.132:0/3777538344 shutdown_connections 2024-03-27T06:18:16.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.151+0000 7f07abe00640 1 -- 172.21.15.132:0/3777538344 wait complete. 2024-03-27T06:18:16.143 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.153+0000 7f07abe00640 1 Processor -- start 2024-03-27T06:18:16.143 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.153+0000 7f07abe00640 1 -- start start 2024-03-27T06:18:16.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac0c7ce0 0x7f07ac158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 0x7f07ac1f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac0c6f00 con 0x7f07ac0c7ce0 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac15acb0 con 0x7f07ac1547a0 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07abe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f07ac15b4b0 con 0x7f07ac073900 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07aaa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07aa000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac0c7ce0 0x7f07ac158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07aaa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59202/0 (socket says 172.21.15.132:59202) 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07aaa00640 1 -- 172.21.15.132:0/1073292948 learned_addr learned my addr 172.21.15.132:0/1073292948 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.154+0000 7f07ab400640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 0x7f07ac1f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 -- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 msgr2=0x7f07ac1f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 0x7f07ac1f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 -- 172.21.15.132:0/1073292948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac0c7ce0 msgr2=0x7f07ac158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:16.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac0c7ce0 0x7f07ac158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 -- 172.21.15.132:0/1073292948 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac1f15e0 con 0x7f07ac073900 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07ab400640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 0x7f07ac1f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07aaa00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f079c00e930 tx=0x7f079c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c017070 con 0x7f07ac073900 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f079c00ca00 con 0x7f07ac073900 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c00cdf0 con 0x7f07ac073900 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.155+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ac1557c0 con 0x7f07ac073900 2024-03-27T06:18:16.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.156+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ac0c45a0 con 0x7f07ac073900 2024-03-27T06:18:16.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.157+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ac0758a0 con 0x7f07ac073900 2024-03-27T06:18:16.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.158+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f079c01a070 con 0x7f07ac073900 2024-03-27T06:18:16.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.159+0000 7f07a3400640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 0x7f078c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:16.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.159+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f079c08b7a0 con 0x7f07ac073900 2024-03-27T06:18:16.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.159+0000 7f07aa000640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 0x7f078c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:16.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.159+0000 7f07aa000640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 0x7f078c06e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0798036000 tx=0x7f079803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:16.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.163+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f07ac1557c0 con 0x7f07ac073900 2024-03-27T06:18:16.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.404+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07ac153400 con 0x7f07ac073900 2024-03-27T06:18:16.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.404+0000 7f07a3400640 1 -- 172.21.15.132:0/1073292948 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6056 (secure 0 0 0) 0x7f079c0596f0 con 0x7f07ac073900 2024-03-27T06:18:16.395 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:16.395 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:14.176322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:resolve","state_seq":18,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273]},"id":1}]} 2024-03-27T06:18:16.395 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 26 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 msgr2=0x7f078c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 0x7f078c06e9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0798036000 tx=0x7f079803b040 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 msgr2=0x7f07ac0c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f079c00e930 tx=0x7f079c00edf0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 shutdown_connections 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f078c06c530 0x7f078c06e9f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac1547a0 0x7f07ac1f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac0c7ce0 0x7f07ac158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 --2- 172.21.15.132:0/1073292948 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f07ac073900 0x7f07ac0c3f80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 >> 172.21.15.132:0/1073292948 conn(0x7f07ac0c9000 msgr2=0x7f07ac151280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:16.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 shutdown_connections 2024-03-27T06:18:16.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:16.407+0000 7f07abe00640 1 -- 172.21.15.132:0/1073292948 wait complete. 2024-03-27T06:18:16.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:18.419 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:18.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.664+0000 7fd728600640 1 Processor -- start 2024-03-27T06:18:18.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.664+0000 7fd728600640 1 -- start start 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.665+0000 7fd728600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd720152950 0x7fd7201590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.665+0000 7fd728600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd720073970 0x7fd72006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.665+0000 7fd728600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72006b630 con 0x7fd720073550 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd728600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd720152d30 con 0x7fd720073970 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd728600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fd72006b8d0 con 0x7fd720152950 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd720152950 0x7fd7201590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd720073550 0x7fd720071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:34818/0 (socket says 172.21.15.132:34818) 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd720152950 0x7fd7201590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:35630/0 (socket says 172.21.15.132:35630) 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd726600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd720073970 0x7fd72006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725200640 1 -- 172.21.15.132:0/4210552339 learned_addr learned my addr 172.21.15.132:0/4210552339 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:18.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 -- 172.21.15.132:0/4210552339 >> v2:172.21.15.132:3300/0 conn(0x7fd720073970 msgr2=0x7fd72006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:18.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 --2- 172.21.15.132:0/4210552339 >> v2:172.21.15.132:3300/0 conn(0x7fd720073970 0x7fd72006cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:18.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 -- 172.21.15.132:0/4210552339 >> v1:172.21.15.134:6789/0 conn(0x7fd720073550 legacy=0x7fd720071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:18.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.666+0000 7fd725c00640 1 -- 172.21.15.132:0/4210552339 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72006bbb0 con 0x7fd720152950 2024-03-27T06:18:18.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.667+0000 7fd725c00640 1 --2- 172.21.15.132:0/4210552339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201590a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7140097a0 tx=0x7fd71402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd6eb886eaac785 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:18.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.668+0000 7fd71fe00640 1 -- 172.21.15.132:0/4210552339 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714037070 con 0x7fd720152950 2024-03-27T06:18:18.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.668+0000 7fd71fe00640 1 -- 172.21.15.132:0/4210552339 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd714030710 con 0x7fd720152950 2024-03-27T06:18:18.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.668+0000 7fd71fe00640 1 -- 172.21.15.132:0/4210552339 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714030ae0 con 0x7fd720152950 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 -- 172.21.15.132:0/4210552339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 msgr2=0x7fd7201590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 --2- 172.21.15.132:0/4210552339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201590a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7140097a0 tx=0x7fd71402b710 comp rx=0 tx=0).stop 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 -- 172.21.15.132:0/4210552339 shutdown_connections 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 --2- 172.21.15.132:0/4210552339 >> v2:172.21.15.132:3300/0 conn(0x7fd720073970 0x7fd72006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 --2- 172.21.15.132:0/4210552339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201590a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 -- 172.21.15.132:0/4210552339 >> 172.21.15.132:0/4210552339 conn(0x7fd7200cc060 msgr2=0x7fd7200cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 -- 172.21.15.132:0/4210552339 shutdown_connections 2024-03-27T06:18:18.659 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.669+0000 7fd728600640 1 -- 172.21.15.132:0/4210552339 wait complete. 2024-03-27T06:18:18.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.671+0000 7fd728600640 1 Processor -- start 2024-03-27T06:18:18.661 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.671+0000 7fd728600640 1 -- start start 2024-03-27T06:18:18.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 0x7fd7201f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd720073970 0x7fd7201f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd72006c450 con 0x7fd720073550 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd72006c170 con 0x7fd720152950 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd728600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd720071880 con 0x7fd720073970 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd725c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 0x7fd7201f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd725200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd720073970 0x7fd7201f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd726600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd725c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 0x7fd7201f0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38528/0 (socket says 172.21.15.132:38528) 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.673+0000 7fd725c00640 1 -- 172.21.15.132:0/1421380573 learned_addr learned my addr 172.21.15.132:0/1421380573 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 -- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd720073970 msgr2=0x7fd7201f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd720073970 0x7fd7201f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:18.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 -- 172.21.15.132:0/1421380573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 msgr2=0x7fd7201f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:18.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 0x7fd7201f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:18.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 -- 172.21.15.132:0/1421380573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd720158770 con 0x7fd720152950 2024-03-27T06:18:18.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd726600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201e5110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7200ccba0 tx=0x7fd71800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:18.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718018020 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd718014480 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718014870 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7200aea70 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.674+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7201f0030 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.675+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd720068840 con 0x7fd720152950 2024-03-27T06:18:18.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.676+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd718005d10 con 0x7fd720152950 2024-03-27T06:18:18.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.677+0000 7fd71d600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 0x7fd70006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:18.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.677+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7fd71808c3d0 con 0x7fd720152950 2024-03-27T06:18:18.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.677+0000 7fd725c00640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 0x7fd70006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:18.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.678+0000 7fd725c00640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 0x7fd70006ea20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd71402b750 tx=0x7fd714008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:18.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:18.682+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd718022030 con 0x7fd720152950 2024-03-27T06:18:19.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.038+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7200c6b30 con 0x7fd720152950 2024-03-27T06:18:19.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.039+0000 7fd71d600640 1 -- 172.21.15.132:0/1421380573 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6056 (secure 0 0 0) 0x7fd718010040 con 0x7fd720152950 2024-03-27T06:18:19.029 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:19.029 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:14.176322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:resolve","state_seq":18,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273]},"id":1}]} 2024-03-27T06:18:19.029 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 26 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.041+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 msgr2=0x7fd70006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 0x7fd70006ea20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd71402b750 tx=0x7fd714008030 comp rx=0 tx=0).stop 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 msgr2=0x7fd7201e5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201e5110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7200ccba0 tx=0x7fd71800ef90 comp rx=0 tx=0).stop 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 shutdown_connections 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd70006c560 0x7fd70006ea20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd720152950 0x7fd7201e5110 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd720073970 0x7fd7201f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:19.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 --2- 172.21.15.132:0/1421380573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd720073550 0x7fd7201f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:19.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 >> 172.21.15.132:0/1421380573 conn(0x7fd7200cc060 msgr2=0x7fd7200cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:19.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 shutdown_connections 2024-03-27T06:18:19.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:19.042+0000 7fd728600640 1 -- 172.21.15.132:0/1421380573 wait complete. 2024-03-27T06:18:19.047 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:21.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:21.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.296+0000 7fa99ec00640 1 Processor -- start 2024-03-27T06:18:21.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.296+0000 7fa99ec00640 1 -- start start 2024-03-27T06:18:21.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa998158290 0x7fa99815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa9980c9c80 0x7fa9980ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa99806ed40 con 0x7fa998158290 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa998158670 con 0x7fa9980c9c80 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.297+0000 7fa99ec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa99806e2f0 con 0x7fa99815b120 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa998158290 0x7fa99815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa99cc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa9980c9c80 0x7fa9980ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59220/0 (socket says 172.21.15.132:59220) 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 -- 172.21.15.132:0/930680823 learned_addr learned my addr 172.21.15.132:0/930680823 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 -- 172.21.15.132:0/930680823 >> v2:172.21.15.134:3300/0 conn(0x7fa9980c9c80 msgr2=0x7fa9980ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 --2- 172.21.15.132:0/930680823 >> v2:172.21.15.134:3300/0 conn(0x7fa9980c9c80 0x7fa9980ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 -- 172.21.15.132:0/930680823 >> v2:172.21.15.132:3300/0 conn(0x7fa998158290 msgr2=0x7fa99815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 --2- 172.21.15.132:0/930680823 >> v2:172.21.15.132:3300/0 conn(0x7fa998158290 0x7fa99815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.298+0000 7fa997400640 1 -- 172.21.15.132:0/930680823 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99806c6d0 con 0x7fa99815b120 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.299+0000 7fa997400640 1 --2- 172.21.15.132:0/930680823 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa99806c540 tx=0x7fa98402b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dac37d4d45bdb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.299+0000 7fa996a00640 1 -- 172.21.15.132:0/930680823 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984035020 con 0x7fa99815b120 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa996a00640 1 -- 172.21.15.132:0/930680823 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98402d040 con 0x7fa99815b120 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa996a00640 1 -- 172.21.15.132:0/930680823 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984031a80 con 0x7fa99815b120 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 -- 172.21.15.132:0/930680823 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 msgr2=0x7fa9980ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:21.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 --2- 172.21.15.132:0/930680823 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa99806c540 tx=0x7fa98402b900 comp rx=0 tx=0).stop 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 -- 172.21.15.132:0/930680823 shutdown_connections 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 --2- 172.21.15.132:0/930680823 >> v2:172.21.15.134:3300/0 conn(0x7fa9980c9c80 0x7fa9980ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 --2- 172.21.15.132:0/930680823 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99815b120 0x7fa9980ce290 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 --2- 172.21.15.132:0/930680823 >> v2:172.21.15.132:3300/0 conn(0x7fa998158290 0x7fa99815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 -- 172.21.15.132:0/930680823 >> 172.21.15.132:0/930680823 conn(0x7fa998067280 msgr2=0x7fa998067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.300+0000 7fa99ec00640 1 -- 172.21.15.132:0/930680823 shutdown_connections 2024-03-27T06:18:21.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.301+0000 7fa99ec00640 1 -- 172.21.15.132:0/930680823 wait complete. 2024-03-27T06:18:21.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.303+0000 7fa99ec00640 1 Processor -- start 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.303+0000 7fa99ec00640 1 -- start start 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 0x7fa9980b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9980cc1e0 0x7fa9980b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa99806e5d0 con 0x7fa9980cc1e0 2024-03-27T06:18:21.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa99806f690 con 0x7fa998158290 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa99ec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa99806f1d0 con 0x7fa9980c9c80 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa997e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa997400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 0x7fa9980b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa997e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:59226/0 (socket says 172.21.15.132:59226) 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.304+0000 7fa997e00640 1 -- 172.21.15.132:0/1743746304 learned_addr learned my addr 172.21.15.132:0/1743746304 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa99cc00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9980cc1e0 0x7fa9980b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 -- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 msgr2=0x7fa9980b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 0x7fa9980b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 -- 172.21.15.132:0/1743746304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9980cc1e0 msgr2=0x7fa9980b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9980cc1e0 0x7fa9980b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 -- 172.21.15.132:0/1743746304 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9980b0100 con 0x7fa9980c9c80 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997400640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 0x7fa9980b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:21.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.305+0000 7fa997e00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa98800e9b0 tx=0x7fa98800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.306+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988017070 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.306+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98800c9a0 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.306+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98800cd90 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.306+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa998152fb0 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.306+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9981f1010 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.307+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa998068840 con 0x7fa9980c9c80 2024-03-27T06:18:21.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.308+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa98801a070 con 0x7fa9980c9c80 2024-03-27T06:18:21.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.309+0000 7fa98fe00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 0x7fa97806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:21.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.309+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7fa98808b830 con 0x7fa9980c9c80 2024-03-27T06:18:21.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.309+0000 7fa997400640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 0x7fa97806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:21.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.309+0000 7fa997400640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 0x7fa97806e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa984036000 tx=0x7fa98403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:21.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.313+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa9880596d0 con 0x7fa9980c9c80 2024-03-27T06:18:21.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.652+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa99806f8f0 con 0x7fa9980c9c80 2024-03-27T06:18:21.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.653+0000 7fa98fe00640 1 -- 172.21.15.132:0/1743746304 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6058 (secure 0 0 0) 0x7fa98805d9d0 con 0x7fa9980c9c80 2024-03-27T06:18:21.643 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:21.644 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:19.208200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718]},"id":1}]} 2024-03-27T06:18:21.644 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 27 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.656+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 msgr2=0x7fa97806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.656+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 0x7fa97806e9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa984036000 tx=0x7fa98403c040 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.656+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 msgr2=0x7fa998151bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.656+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa98800e9b0 tx=0x7fa98800ee70 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 shutdown_connections 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9980cc1e0 0x7fa9980b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97806c530 0x7fa97806e9f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa998158290 0x7fa9980b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 --2- 172.21.15.132:0/1743746304 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa9980c9c80 0x7fa998151bc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:21.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 >> 172.21.15.132:0/1743746304 conn(0x7fa998067280 msgr2=0x7fa9980c3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:21.648 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 shutdown_connections 2024-03-27T06:18:21.648 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:21.657+0000 7fa99ec00640 1 -- 172.21.15.132:0/1743746304 wait complete. 2024-03-27T06:18:21.665 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:23.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:23.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.903+0000 7fbd3d800640 1 Processor -- start 2024-03-27T06:18:23.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.903+0000 7fbd3d800640 1 -- start start 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.905+0000 7fbd3d800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 0x7fbd3815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.905+0000 7fbd3d800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fbd3806b520 0x7fbd3806b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.905+0000 7fbd3d800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd380c1120 con 0x7fbd38158f10 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.905+0000 7fbd3d800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fbd3815fc20 con 0x7fbd3806b520 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.905+0000 7fbd3d800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fbd380c13e0 con 0x7fbd3806daf0 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd36a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbd38158f10 0x7fbd3806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39110/0 (socket says 172.21.15.132:39110) 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd36a00640 1 -- 172.21.15.132:0/2711571967 learned_addr learned my addr 172.21.15.132:0/2711571967 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd37400640 1 --2- 172.21.15.132:0/2711571967 >> v2:172.21.15.132:3300/0 conn(0x7fbd3806b520 0x7fbd3806b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd2e000640 1 --2- 172.21.15.132:0/2711571967 >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 0x7fbd3815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd36000640 1 -- 172.21.15.132:0/2711571967 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344618378 0 0) 0x7fbd380c1120 con 0x7fbd38158f10 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd36000640 1 -- 172.21.15.132:0/2711571967 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd1c003520 con 0x7fbd38158f10 2024-03-27T06:18:23.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd37400640 1 -- 172.21.15.132:0/2711571967 >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 msgr2=0x7fbd3815f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:23.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd37400640 1 --2- 172.21.15.132:0/2711571967 >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 0x7fbd3815f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd37400640 1 -- 172.21.15.132:0/2711571967 >> v1:172.21.15.134:6789/0 conn(0x7fbd38158f10 legacy=0x7fbd3806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:23.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd37400640 1 -- 172.21.15.132:0/2711571967 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd380c16c0 con 0x7fbd3806b520 2024-03-27T06:18:23.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.906+0000 7fbd2e000640 1 --2- 172.21.15.132:0/2711571967 >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 0x7fbd3815f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.907+0000 7fbd37400640 1 --2- 172.21.15.132:0/2711571967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806b520 0x7fbd3806b950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd380c9c00 tx=0x7fbd3002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea0a10c0cd8542f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd36000640 1 -- 172.21.15.132:0/2711571967 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30039070 con 0x7fbd3806b520 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd36000640 1 -- 172.21.15.132:0/2711571967 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3002cc70 con 0x7fbd3806b520 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd36000640 1 -- 172.21.15.132:0/2711571967 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30034660 con 0x7fbd3806b520 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 -- 172.21.15.132:0/2711571967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806b520 msgr2=0x7fbd3806b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 --2- 172.21.15.132:0/2711571967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806b520 0x7fbd3806b950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd380c9c00 tx=0x7fbd3002bec0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 -- 172.21.15.132:0/2711571967 shutdown_connections 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 --2- 172.21.15.132:0/2711571967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806b520 0x7fbd3806b950 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 --2- 172.21.15.132:0/2711571967 >> v2:172.21.15.134:3300/0 conn(0x7fbd3806daf0 0x7fbd3815f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 -- 172.21.15.132:0/2711571967 >> 172.21.15.132:0/2711571967 conn(0x7fbd380cc060 msgr2=0x7fbd380cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:23.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 -- 172.21.15.132:0/2711571967 shutdown_connections 2024-03-27T06:18:23.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.908+0000 7fbd3d800640 1 -- 172.21.15.132:0/2711571967 wait complete. 2024-03-27T06:18:23.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.910+0000 7fbd3d800640 1 Processor -- start 2024-03-27T06:18:23.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.910+0000 7fbd3d800640 1 -- start start 2024-03-27T06:18:23.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd3806b520 0x7fbd3815df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806daf0 0x7fbd381ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380c1f60 con 0x7fbd3806daf0 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380c1c80 con 0x7fbd38158f10 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.911+0000 7fbd3d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3806cec0 con 0x7fbd3806b520 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd36a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd3806b520 0x7fbd3815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd2e000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806daf0 0x7fbd381ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39020/0 (socket says 172.21.15.132:39020) 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 -- 172.21.15.132:0/1054023098 learned_addr learned my addr 172.21.15.132:0/1054023098 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 -- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd3806b520 msgr2=0x7fbd3815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:23.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd3806b520 0x7fbd3815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 -- 172.21.15.132:0/1054023098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806daf0 msgr2=0x7fbd381ea7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806daf0 0x7fbd381ea7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 -- 172.21.15.132:0/1054023098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381f4ad0 con 0x7fbd38158f10 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd37400640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009e80 tx=0x7fbd30004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30039050 con 0x7fbd38158f10 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd30045070 con 0x7fbd38158f10 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.912+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd381f4db0 con 0x7fbd38158f10 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.913+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30009450 con 0x7fbd38158f10 2024-03-27T06:18:23.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.913+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd381f5370 con 0x7fbd38158f10 2024-03-27T06:18:23.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.914+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd381f55b0 con 0x7fbd38158f10 2024-03-27T06:18:23.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.914+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fbd3003e850 con 0x7fbd38158f10 2024-03-27T06:18:23.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.915+0000 7fbd2f400640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 0x7fbd1406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:23.906 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.915+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7fbd300ae4c0 con 0x7fbd38158f10 2024-03-27T06:18:23.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.916+0000 7fbd36a00640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 0x7fbd1406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:23.907 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.916+0000 7fbd36a00640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 0x7fbd1406e9f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd28005880 tx=0x7fbd28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:23.920+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fbd3002ce20 con 0x7fbd38158f10 2024-03-27T06:18:24.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.161+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd38069560 con 0x7fbd38158f10 2024-03-27T06:18:24.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.162+0000 7fbd2f400640 1 -- 172.21.15.132:0/1054023098 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6058 (secure 0 0 0) 0x7fbd38069560 con 0x7fbd38158f10 2024-03-27T06:18:24.153 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:24.153 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:19.208200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718]},"id":1}]} 2024-03-27T06:18:24.154 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 27 2024-03-27T06:18:24.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 msgr2=0x7fbd1406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:24.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 0x7fbd1406e9f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd28005880 tx=0x7fbd28009040 comp rx=0 tx=0).stop 2024-03-27T06:18:24.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 msgr2=0x7fbd3814e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009e80 tx=0x7fbd30004a40 comp rx=0 tx=0).stop 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 shutdown_connections 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fbd1406c530 0x7fbd1406e9f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd38158f10 0x7fbd3814e470 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd3806daf0 0x7fbd381ea7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 --2- 172.21.15.132:0/1054023098 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fbd3806b520 0x7fbd3815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 >> 172.21.15.132:0/1054023098 conn(0x7fbd380cc060 msgr2=0x7fbd380cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 shutdown_connections 2024-03-27T06:18:24.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:24.164+0000 7fbd3d800640 1 -- 172.21.15.132:0/1054023098 wait complete. 2024-03-27T06:18:24.175 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4273": { "addr": "172.21.15.134:6839/3802035585", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 3802035585, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 3802035585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 21 }, "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:18:19.208200+0000", "qdb_cluster": [ 4273, 4718 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273, "mds_1": 4817, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:18:26.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:26.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.392+0000 7f0634a00640 1 Processor -- start 2024-03-27T06:18:26.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.392+0000 7f0634a00640 1 -- start start 2024-03-27T06:18:26.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f0634a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f06300d0970 0x7f063015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f0634a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f06300cb440 0x7f06300c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f0634a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06300cb9f0 con 0x7f06300d0d50 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f0634a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f063015e2e0 con 0x7f06300cb440 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f0634a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f063015e5c0 con 0x7f06300d0970 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f06300d0970 0x7f063015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f06300d0970 0x7f063015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39034/0 (socket says 172.21.15.132:39034) 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 -- 172.21.15.132:0/4121013959 learned_addr learned my addr 172.21.15.132:0/4121013959 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062e800640 1 --2- 172.21.15.132:0/4121013959 >> v2:172.21.15.132:3300/0 conn(0x7f06300cb440 0x7f06300c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 -- 172.21.15.132:0/4121013959 >> v2:172.21.15.132:3300/0 conn(0x7f06300cb440 msgr2=0x7f06300c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 --2- 172.21.15.132:0/4121013959 >> v2:172.21.15.132:3300/0 conn(0x7f06300cb440 0x7f06300c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 -- 172.21.15.132:0/4121013959 >> v1:172.21.15.132:6789/0 conn(0x7f06300d0d50 legacy=0x7f0630154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.394+0000 7f062de00640 1 -- 172.21.15.132:0/4121013959 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063015e8a0 con 0x7f06300d0970 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f062de00640 1 --2- 172.21.15.132:0/4121013959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0970 0x7f063015b6d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f063008d3e0 tx=0x7f062002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21dd6447144ff78f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f062ca00640 1 -- 172.21.15.132:0/4121013959 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620034070 con 0x7f06300d0970 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f062ca00640 1 -- 172.21.15.132:0/4121013959 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0620030860 con 0x7f06300d0970 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f062ca00640 1 -- 172.21.15.132:0/4121013959 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620030c30 con 0x7f06300d0970 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 -- 172.21.15.132:0/4121013959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0970 msgr2=0x7f063015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 --2- 172.21.15.132:0/4121013959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0970 0x7f063015b6d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f063008d3e0 tx=0x7f062002b9a0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 -- 172.21.15.132:0/4121013959 shutdown_connections 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 --2- 172.21.15.132:0/4121013959 >> v2:172.21.15.132:3300/0 conn(0x7f06300cb440 0x7f06300c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 --2- 172.21.15.132:0/4121013959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0970 0x7f063015b6d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 -- 172.21.15.132:0/4121013959 >> 172.21.15.132:0/4121013959 conn(0x7f063006dbd0 msgr2=0x7f063006dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.396+0000 7f0634a00640 1 -- 172.21.15.132:0/4121013959 shutdown_connections 2024-03-27T06:18:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.397+0000 7f0634a00640 1 -- 172.21.15.132:0/4121013959 wait complete. 2024-03-27T06:18:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.398+0000 7f0634a00640 1 Processor -- start 2024-03-27T06:18:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.398+0000 7f0634a00640 1 -- start start 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f06300d0970 0x7f06300cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 0x7f06300ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0630155330 con 0x7f06300cb440 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f06301551b0 con 0x7f06300d0d50 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f0634a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f06300cc7f0 con 0x7f06300d0970 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f062de00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f062de00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:47022/0 (socket says 172.21.15.132:47022) 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f062de00640 1 -- 172.21.15.132:0/1890235620 learned_addr learned my addr 172.21.15.132:0/1890235620 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f062d400640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f06300d0970 0x7f06300cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.399+0000 7f062e800640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 0x7f06300ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 -- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f06300d0970 msgr2=0x7f06300cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f06300d0970 0x7f06300cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 -- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 msgr2=0x7f06300ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 0x7f06300ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 -- 172.21.15.132:0/1890235620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06301f0470 con 0x7f06300cb440 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062e800640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 0x7f06300ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.400+0000 7f062de00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f062002b9e0 tx=0x7f0620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.401+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620048070 con 0x7f06300cb440 2024-03-27T06:18:26.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.401+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0620030660 con 0x7f06300cb440 2024-03-27T06:18:26.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.401+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06301e62b0 con 0x7f06300cb440 2024-03-27T06:18:26.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.401+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620034040 con 0x7f06300cb440 2024-03-27T06:18:26.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.401+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06301e6870 con 0x7f06300cb440 2024-03-27T06:18:26.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.402+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0630068840 con 0x7f06300cb440 2024-03-27T06:18:26.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.403+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f062004b070 con 0x7f06300cb440 2024-03-27T06:18:26.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.404+0000 7f0626000640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 0x7f061006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:26.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.404+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f06200ad650 con 0x7f06300cb440 2024-03-27T06:18:26.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.404+0000 7f062d400640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 0x7f061006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:26.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.405+0000 7f062d400640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 0x7f061006ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f06300cd760 tx=0x7f061c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:26.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.409+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f062007b4f0 con 0x7f06300cb440 2024-03-27T06:18:26.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.648+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06300cbff0 con 0x7f06300cb440 2024-03-27T06:18:26.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.649+0000 7f0626000640 1 -- 172.21.15.132:0/1890235620 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6055 (secure 0 0 0) 0x7f062007f7f0 con 0x7f06300cb440 2024-03-27T06:18:26.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:26.641 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:24.247559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273]},"id":1}]} 2024-03-27T06:18:26.641 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 28 2024-03-27T06:18:26.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 msgr2=0x7f061006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 0x7f061006ea20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f06300cd760 tx=0x7f061c009040 comp rx=0 tx=0).stop 2024-03-27T06:18:26.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 msgr2=0x7f063014f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f062002b9e0 tx=0x7f0620004730 comp rx=0 tx=0).stop 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 shutdown_connections 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f061006c560 0x7f061006ea20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06300d0d50 0x7f06300ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f06300d0970 0x7f06300cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 --2- 172.21.15.132:0/1890235620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06300cb440 0x7f063014f830 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 >> 172.21.15.132:0/1890235620 conn(0x7f063006dbd0 msgr2=0x7f0630154a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 shutdown_connections 2024-03-27T06:18:26.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:26.652+0000 7f0634a00640 1 -- 172.21.15.132:0/1890235620 wait complete. 2024-03-27T06:18:26.660 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:28.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:28.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.879+0000 7f6542800640 1 Processor -- start 2024-03-27T06:18:28.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.879+0000 7f6542800640 1 -- start start 2024-03-27T06:18:28.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f6542800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f6542800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 0x7f653c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f6542800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653c15b790 con 0x7f653c0b11e0 2024-03-27T06:18:28.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f6542800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f653c0c1080 con 0x7f653c15aa00 2024-03-27T06:18:28.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f6542800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f653c0c1c00 con 0x7f653c06b520 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f653b400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f653c0b11e0 0x7f653c0c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:39118/0 (socket says 172.21.15.132:39118) 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f653aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f653b400640 1 -- 172.21.15.132:0/3317625244 learned_addr learned my addr 172.21.15.132:0/3317625244 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f653be00640 1 --2- 172.21.15.132:0/3317625244 >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 0x7f653c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.883+0000 7f653aa00640 1 -- 172.21.15.132:0/3317625244 >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 msgr2=0x7f653c15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.882+0000 7f653a000640 1 -- 172.21.15.132:0/3317625244 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432428484 0 0) 0x7f653c15b790 con 0x7f653c0b11e0 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.883+0000 7f653aa00640 1 --2- 172.21.15.132:0/3317625244 >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 0x7f653c15ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.883+0000 7f653aa00640 1 -- 172.21.15.132:0/3317625244 >> v1:172.21.15.134:6789/0 conn(0x7f653c0b11e0 legacy=0x7f653c0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.883+0000 7f653aa00640 1 -- 172.21.15.132:0/3317625244 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c0c12d0 con 0x7f653c06b520 2024-03-27T06:18:28.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.883+0000 7f653be00640 1 --2- 172.21.15.132:0/3317625244 >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 0x7f653c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:28.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.884+0000 7f653aa00640 1 --2- 172.21.15.132:0/3317625244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c06b960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65280097a0 tx=0x7f652802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8fa0daeed4c531c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.884+0000 7f653a000640 1 -- 172.21.15.132:0/3317625244 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528034070 con 0x7f653c06b520 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.884+0000 7f653a000640 1 -- 172.21.15.132:0/3317625244 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6528030800 con 0x7f653c06b520 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.884+0000 7f653a000640 1 -- 172.21.15.132:0/3317625244 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528030bd0 con 0x7f653c06b520 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 -- 172.21.15.132:0/3317625244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 msgr2=0x7f653c06b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 --2- 172.21.15.132:0/3317625244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c06b960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65280097a0 tx=0x7f652802b940 comp rx=0 tx=0).stop 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 -- 172.21.15.132:0/3317625244 shutdown_connections 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 --2- 172.21.15.132:0/3317625244 >> v2:172.21.15.134:3300/0 conn(0x7f653c15aa00 0x7f653c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 --2- 172.21.15.132:0/3317625244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c06b960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 -- 172.21.15.132:0/3317625244 >> 172.21.15.132:0/3317625244 conn(0x7f653c0cc060 msgr2=0x7f653c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 -- 172.21.15.132:0/3317625244 shutdown_connections 2024-03-27T06:18:28.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.885+0000 7f6542800640 1 -- 172.21.15.132:0/3317625244 wait complete. 2024-03-27T06:18:28.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.888+0000 7f6542800640 1 Processor -- start 2024-03-27T06:18:28.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.888+0000 7f6542800640 1 -- start start 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f653c15aa00 0x7f653c1e5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0b15c0 con 0x7f653c15aa00 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0c1890 con 0x7f653c0b11e0 2024-03-27T06:18:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f6542800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f653c0c2090 con 0x7f653c06b520 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39054/0 (socket says 172.21.15.132:39054) 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653aa00640 1 -- 172.21.15.132:0/270064261 learned_addr learned my addr 172.21.15.132:0/270064261 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:45630/0 (socket says 172.21.15.132:45630) 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653be00640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f653c15aa00 0x7f653c1e5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 -- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 msgr2=0x7f653c1e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 -- 172.21.15.132:0/270064261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f653c15aa00 msgr2=0x7f653c1e5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f653c15aa00 0x7f653c1e5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653b400640 1 -- 172.21.15.132:0/270064261 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c1e4d70 con 0x7f653c06b520 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.890+0000 7f653aa00640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f653b400640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f652c00c3e0 tx=0x7f652c00c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0074e0 con 0x7f653c06b520 2024-03-27T06:18:28.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f652c007680 con 0x7f653c06b520 2024-03-27T06:18:28.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c156020 con 0x7f653c06b520 2024-03-27T06:18:28.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c004a20 con 0x7f653c06b520 2024-03-27T06:18:28.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.891+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c1e5010 con 0x7f653c06b520 2024-03-27T06:18:28.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.892+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c068840 con 0x7f653c06b520 2024-03-27T06:18:28.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.894+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f652c011070 con 0x7f653c06b520 2024-03-27T06:18:28.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.895+0000 7f6533400640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 0x7f651c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:28.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.895+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f652c08b270 con 0x7f653c06b520 2024-03-27T06:18:28.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.895+0000 7f653aa00640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 0x7f651c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:28.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.896+0000 7f653aa00640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 0x7f651c06ea20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f652802b980 tx=0x7f652803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:28.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:28.899+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f652c059800 con 0x7f653c06b520 2024-03-27T06:18:29.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.140+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f653c06e9c0 con 0x7f653c06b520 2024-03-27T06:18:29.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.141+0000 7f6533400640 1 -- 172.21.15.132:0/270064261 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6055 (secure 0 0 0) 0x7f652c059100 con 0x7f653c06b520 2024-03-27T06:18:29.133 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:29.133 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:24.247559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4718,4273]},"id":1}]} 2024-03-27T06:18:29.133 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 28 2024-03-27T06:18:29.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 msgr2=0x7f651c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:29.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 0x7f651c06ea20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f652802b980 tx=0x7f652803b040 comp rx=0 tx=0).stop 2024-03-27T06:18:29.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 msgr2=0x7f653c0c3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:29.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f652c00c3e0 tx=0x7f652c00c7e0 comp rx=0 tx=0).stop 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 shutdown_connections 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f653c15aa00 0x7f653c1e5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f651c06c560 0x7f651c06ea20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f653c0b11e0 0x7f653c1e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 --2- 172.21.15.132:0/270064261 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f653c06b520 0x7f653c0c3520 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 >> 172.21.15.132:0/270064261 conn(0x7f653c0cc060 msgr2=0x7f653c0c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:29.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 shutdown_connections 2024-03-27T06:18:29.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:29.143+0000 7f6542800640 1 -- 172.21.15.132:0/270064261 wait complete. 2024-03-27T06:18:29.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:18:29.277 INFO:tasks.workunit.client.0.smithi132.stderr:+ set -e 2024-03-27T06:18:29.277 INFO:tasks.workunit.client.0.smithi132.stderr:++ seq 1 20 2024-03-27T06:18:29.279 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:29.281 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:29.284 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:29.285 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:29.317 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:29.346 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:29.349 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:29.520 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:29.572 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:29.572 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:29.572 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:29.572 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:29.572 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:29.573 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:29.573 INFO:tasks.workunit.client.0.smithi132.stdout: 49194 ? 00:00:00 ps 2024-03-27T06:18:29.574 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:31.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:31.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.385+0000 7f2211a00640 1 Processor -- start 2024-03-27T06:18:31.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.385+0000 7f2211a00640 1 -- start start 2024-03-27T06:18:31.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.386+0000 7f2211a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f220c152950 0x7f220c1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.386+0000 7f2211a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 0x7f220c06cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.386+0000 7f2211a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220c06b630 con 0x7f220c073550 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.386+0000 7f2211a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f220c152d30 con 0x7f220c152950 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.386+0000 7f2211a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f220c06b8d0 con 0x7f220c073970 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f220c152950 0x7f220c1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 0x7f220c06cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f220c152950 0x7f220c1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:39056/0 (socket says 172.21.15.132:39056) 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 -- 172.21.15.132:0/3648447757 learned_addr learned my addr 172.21.15.132:0/3648447757 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:31.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 -- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 msgr2=0x7f220c06cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:31.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 0x7f220c06cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 -- 172.21.15.132:0/3648447757 >> v1:172.21.15.132:6789/0 conn(0x7f220c073550 legacy=0x7f220c071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220aa00640 1 -- 172.21.15.132:0/3648447757 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c06bbb0 con 0x7f220c152950 2024-03-27T06:18:31.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.387+0000 7f220b400640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 0x7f220c06cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:31.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.389+0000 7f220aa00640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c1590a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009870 tx=0x7f21fc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b599d65bb72bd210 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:31.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.389+0000 7f2209600640 1 -- 172.21.15.132:0/3648447757 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc034070 con 0x7f220c152950 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.389+0000 7f2209600640 1 -- 172.21.15.132:0/3648447757 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21fc030840 con 0x7f220c152950 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.389+0000 7f2209600640 1 -- 172.21.15.132:0/3648447757 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc030c10 con 0x7f220c152950 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 -- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 msgr2=0x7f220c1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c1590a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009870 tx=0x7f21fc02b980 comp rx=0 tx=0).stop 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 -- 172.21.15.132:0/3648447757 shutdown_connections 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073970 0x7f220c06cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 --2- 172.21.15.132:0/3648447757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c1590a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 -- 172.21.15.132:0/3648447757 >> 172.21.15.132:0/3648447757 conn(0x7f220c0cc060 msgr2=0x7f220c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 -- 172.21.15.132:0/3648447757 shutdown_connections 2024-03-27T06:18:31.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.390+0000 7f2211a00640 1 -- 172.21.15.132:0/3648447757 wait complete. 2024-03-27T06:18:31.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.391+0000 7f2211a00640 1 Processor -- start 2024-03-27T06:18:31.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 -- start start 2024-03-27T06:18:31.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220c073970 0x7f220c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c06d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f220c06c450 con 0x7f220c073970 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f220c06c170 con 0x7f220c152950 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.392+0000 7f2211a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f220c071880 con 0x7f220c073550 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:45650/0 (socket says 172.21.15.132:45650) 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 -- 172.21.15.132:0/1396925325 learned_addr learned my addr 172.21.15.132:0/1396925325 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220a000640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220c073970 0x7f220c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220b400640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c06d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 -- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 msgr2=0x7f220c06d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c06d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 -- 172.21.15.132:0/1396925325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220c073970 msgr2=0x7f220c154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220c073970 0x7f220c154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 -- 172.21.15.132:0/1396925325 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c1547c0 con 0x7f220c073550 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.393+0000 7f220aa00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f21fc000c00 tx=0x7f21fc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:31.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.394+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc034040 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.394+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21fc039d90 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.394+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc043610 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.394+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c0c5390 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.394+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c1ee840 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.395+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f220c1ee670 con 0x7f220c073550 2024-03-27T06:18:31.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.396+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f21fc04a060 con 0x7f220c073550 2024-03-27T06:18:31.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.397+0000 7f2202a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 0x7f21ec06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:31.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.397+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f21fc0ad6b0 con 0x7f220c073550 2024-03-27T06:18:31.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.397+0000 7f220a000640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 0x7f21ec06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:31.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.398+0000 7f220a000640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 0x7f21ec06ea40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f220c0c9040 tx=0x7f21f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:31.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.401+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f21fc03e070 con 0x7f220c073550 2024-03-27T06:18:31.392 INFO:tasks.ceph.mds.f.smithi134.stderr:2024-03-27T06:18:31.402+0000 7f8511512640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-27T06:18:31.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.638+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f220c0c64f0 con 0x7f220c073550 2024-03-27T06:18:31.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.639+0000 7f2202a00640 1 -- 172.21.15.132:0/1396925325 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6060 (secure 0 0 0) 0x7f220c0c64f0 con 0x7f220c073550 2024-03-27T06:18:31.629 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:31.629 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718,4817]},"id":1}]} 2024-03-27T06:18:31.629 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 29 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.642+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 msgr2=0x7f21ec06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.642+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 0x7f21ec06ea40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f220c0c9040 tx=0x7f21f8009040 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 msgr2=0x7f220c1f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f21fc000c00 tx=0x7f21fc0046c0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 shutdown_connections 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f21ec06c580 0x7f21ec06ea40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c152950 0x7f220c06d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220c073970 0x7f220c154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 --2- 172.21.15.132:0/1396925325 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f220c073550 0x7f220c1f2620 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 >> 172.21.15.132:0/1396925325 conn(0x7f220c0cc060 msgr2=0x7f220c0cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:31.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 shutdown_connections 2024-03-27T06:18:31.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:31.643+0000 7f2211a00640 1 -- 172.21.15.132:0/1396925325 wait complete. 2024-03-27T06:18:31.651 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=4270 2024-03-27T06:18:31.651 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2024-03-27T06:18:34.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:18:34.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:18:34.505 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:34.512 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:34.524 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:34.524 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:34.528 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:34.529 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:34.560 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:34.588 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:34.590 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:34.712 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:34.748 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:34.749 INFO:tasks.workunit.client.0.smithi132.stdout: 49267 ? 00:00:00 ps 2024-03-27T06:18:34.750 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:34.770 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:34.776 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:34.785 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:34.785 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:34.792 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:34.793 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:34.825 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:34.854 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:34.856 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:34.928 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:34.963 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:34.964 INFO:tasks.workunit.client.0.smithi132.stdout: 49277 ? 00:00:00 ps 2024-03-27T06:18:34.965 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:34.982 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:34.988 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:34.998 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:34.998 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.003 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.003 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:35.035 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:35.064 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:35.066 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:35.135 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:35.192 INFO:tasks.workunit.client.0.smithi132.stdout: 49287 ? 00:00:00 ps 2024-03-27T06:18:35.193 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:35.208 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:35.213 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:35.225 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:35.225 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.229 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.229 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:35.261 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:35.291 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:35.293 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:35.360 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:35.397 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:35.397 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:35.397 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:35.398 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:35.398 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:35.398 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:35.398 INFO:tasks.workunit.client.0.smithi132.stdout: 49297 ? 00:00:00 ps 2024-03-27T06:18:35.398 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:35.415 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:35.420 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:35.429 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:35.429 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.433 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.433 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:35.464 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:35.493 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:35.495 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:35.560 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:35.596 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:35.596 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:35.596 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:35.596 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:35.597 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:35.597 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:35.597 INFO:tasks.workunit.client.0.smithi132.stdout: 49307 ? 00:00:00 ps 2024-03-27T06:18:35.597 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:35.614 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:35.618 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:35.628 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:35.628 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.632 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.633 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:35.652 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-03-27T06:18:35.652 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-27T06:18:35.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-27T06:18:35.653 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:35.655 INFO:tasks.ceph.mds.b:Started 2024-03-27T06:18:35.655 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:35.672 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:35.674 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:35.731 INFO:tasks.ceph.mds.b.smithi134.stdout:starting mds.b at 2024-03-27T06:18:35.755 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:35.788 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:35.788 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:35.788 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:35.788 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:35.789 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:35.789 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:35.789 INFO:tasks.workunit.client.0.smithi132.stdout: 49337 ? 00:00:00 ps 2024-03-27T06:18:35.790 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:35.804 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:35.809 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:35.821 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:35.821 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.825 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.825 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:35.847 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:35.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.861+0000 7f12b9200640 1 Processor -- start 2024-03-27T06:18:35.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.861+0000 7f12b9200640 1 -- start start 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.862+0000 7f12b9200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b4152950 0x7f12b41590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.862+0000 7f12b9200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f12b4073970 0x7f12b406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b9200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b406b630 con 0x7f12b4073550 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b9200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f12b4152d30 con 0x7f12b4073970 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b9200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f12b406b8d0 con 0x7f12b4152950 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b4152950 0x7f12b41590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b2800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f12b4073970 0x7f12b406cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12b4152950 0x7f12b41590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:42310/0 (socket says 172.21.15.132:42310) 2024-03-27T06:18:35.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 -- 172.21.15.132:0/219666575 learned_addr learned my addr 172.21.15.132:0/219666575 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:35.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 -- 172.21.15.132:0/219666575 >> v2:172.21.15.132:3300/0 conn(0x7f12b4073970 msgr2=0x7f12b406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:35.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 --2- 172.21.15.132:0/219666575 >> v2:172.21.15.132:3300/0 conn(0x7f12b4073970 0x7f12b406cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 -- 172.21.15.132:0/219666575 >> v1:172.21.15.132:6789/0 conn(0x7f12b4073550 legacy=0x7f12b4071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:35.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.863+0000 7f12b1e00640 1 -- 172.21.15.132:0/219666575 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b406bbb0 con 0x7f12b4152950 2024-03-27T06:18:35.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.865+0000 7f12b1e00640 1 --2- 172.21.15.132:0/219666575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4152950 0x7f12b41590a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009870 tx=0x7f12a402b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11e679d71cf89129 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:35.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.865+0000 7f12b1400640 1 -- 172.21.15.132:0/219666575 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4034070 con 0x7f12b4152950 2024-03-27T06:18:35.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.865+0000 7f12b1400640 1 -- 172.21.15.132:0/219666575 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a4030860 con 0x7f12b4152950 2024-03-27T06:18:35.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.865+0000 7f12b1400640 1 -- 172.21.15.132:0/219666575 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4030c90 con 0x7f12b4152950 2024-03-27T06:18:35.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 -- 172.21.15.132:0/219666575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4152950 msgr2=0x7f12b41590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:35.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 --2- 172.21.15.132:0/219666575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4152950 0x7f12b41590a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009870 tx=0x7f12a402b9c0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 -- 172.21.15.132:0/219666575 shutdown_connections 2024-03-27T06:18:35.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 --2- 172.21.15.132:0/219666575 >> v2:172.21.15.132:3300/0 conn(0x7f12b4073970 0x7f12b406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 --2- 172.21.15.132:0/219666575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4152950 0x7f12b41590a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 -- 172.21.15.132:0/219666575 >> 172.21.15.132:0/219666575 conn(0x7f12b40cc060 msgr2=0x7f12b40cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:35.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 -- 172.21.15.132:0/219666575 shutdown_connections 2024-03-27T06:18:35.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.866+0000 7f12b9200640 1 -- 172.21.15.132:0/219666575 wait complete. 2024-03-27T06:18:35.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.869+0000 7f12b9200640 1 Processor -- start 2024-03-27T06:18:35.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.869+0000 7f12b9200640 1 -- start start 2024-03-27T06:18:35.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b4073970 0x7f12b40c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b4152950 0x7f12b4157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12b406c450 con 0x7f12b4073970 2024-03-27T06:18:35.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12b406c170 con 0x7f12b4073550 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.870+0000 7f12b9200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f12b4071880 con 0x7f12b4152950 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:42312/0 (socket says 172.21.15.132:42312) 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12abe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b4073970 0x7f12b40c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 -- 172.21.15.132:0/811548105 learned_addr learned my addr 172.21.15.132:0/811548105 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b2800640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b4152950 0x7f12b4157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 -- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b4152950 msgr2=0x7f12b4157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b4152950 0x7f12b4157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 -- 172.21.15.132:0/811548105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b4073970 msgr2=0x7f12b40c39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:35.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b4073970 0x7f12b40c39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:35.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.871+0000 7f12b1e00640 1 -- 172.21.15.132:0/811548105 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b41f4530 con 0x7f12b4073550 2024-03-27T06:18:35.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12b1e00640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f12a40090b0 tx=0x7f12a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:35.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4034040 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a4003820 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b41f47f0 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4003bf0 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.872+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b41f0aa0 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.873+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b40c63f0 con 0x7f12b4073550 2024-03-27T06:18:35.864 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:35.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.875+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f12a404a060 con 0x7f12b4073550 2024-03-27T06:18:35.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.876+0000 7f12aa000640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 0x7f129006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:35.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.876+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f12a40ad6d0 con 0x7f12b4073550 2024-03-27T06:18:35.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.876+0000 7f12abe00640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 0x7f129006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:35.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.877+0000 7f12abe00640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 0x7f129006ea20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f12b40b1150 tx=0x7f129c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:35.867 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:35.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:35.880+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f12b40c63f0 con 0x7f12b4073550 2024-03-27T06:18:35.921 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:35.953 INFO:tasks.workunit.client.0.smithi132.stdout: 49367 ? 00:00:00 ps 2024-03-27T06:18:35.954 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:35.969 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:35.974 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:35.983 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:35.983 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:35.989 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:35.989 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.009 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.025 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.029 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:36.104 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:36.137 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:36.137 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:36.137 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:36.138 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:36.138 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:36.138 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:36.138 INFO:tasks.workunit.client.0.smithi132.stdout: 49377 ? 00:00:00 ps 2024-03-27T06:18:36.138 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:36.155 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:36.160 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:36.169 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:36.169 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:36.174 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:36.174 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.193 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.219+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12b4068840 con 0x7f12b4073550 2024-03-27T06:18:36.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.220+0000 7f12aa000640 1 -- 172.21.15.132:0/811548105 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6060 (secure 0 0 0) 0x7f12a403e070 con 0x7f12b4073550 2024-03-27T06:18:36.210 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:36.210 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273,4718,4817]},"id":1}]} 2024-03-27T06:18:36.211 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 29 2024-03-27T06:18:36.211 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 msgr2=0x7f129006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 0x7f129006ea20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f12b40b1150 tx=0x7f129c009040 comp rx=0 tx=0).stop 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 msgr2=0x7f12b4151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f12a40090b0 tx=0x7f12a40046c0 comp rx=0 tx=0).stop 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 shutdown_connections 2024-03-27T06:18:36.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f12b4152950 0x7f12b4157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12b4073970 0x7f12b40c39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f129006c560 0x7f129006ea20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 --2- 172.21.15.132:0/811548105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12b4073550 0x7f12b4151b40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 >> 172.21.15.132:0/811548105 conn(0x7f12b40cc060 msgr2=0x7f12b40c1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:36.214 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 shutdown_connections 2024-03-27T06:18:36.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:36.223+0000 7f12b9200640 1 -- 172.21.15.132:0/811548105 wait complete. 2024-03-27T06:18:36.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-27T06:18:36.287 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:36.323 INFO:tasks.workunit.client.0.smithi132.stdout: 49389 ? 00:00:00 ps 2024-03-27T06:18:36.324 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:36.340 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:36.345 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:36.354 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:36.354 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:36.358 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:36.359 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.379 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.398 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.401 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:36.498 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:36.534 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:36.534 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:36.534 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:36.534 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:36.535 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:36.535 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:36.535 INFO:tasks.workunit.client.0.smithi132.stdout: 49399 ? 00:00:00 ps 2024-03-27T06:18:36.535 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:36.553 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:36.558 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:36.568 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:36.568 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:36.572 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:36.573 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.594 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.611 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.613 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:36.671 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:36.706 INFO:tasks.workunit.client.0.smithi132.stdout: 49409 ? 00:00:00 ps 2024-03-27T06:18:36.707 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:36.726 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:36.730 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:36.738 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:36.738 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:36.742 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:36.742 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.762 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.781 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.783 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:36.854 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:36.889 INFO:tasks.workunit.client.0.smithi132.stdout: 49419 ? 00:00:00 ps 2024-03-27T06:18:36.890 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:36.907 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:36.912 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:36.922 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:36.922 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:36.928 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:36.928 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:36.951 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:36.970 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:36.972 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:37.029 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:37.065 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:37.065 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:37.065 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:37.065 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:37.065 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:37.066 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:37.066 INFO:tasks.workunit.client.0.smithi132.stdout: 49429 ? 00:00:00 ps 2024-03-27T06:18:37.067 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:37.082 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:37.086 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:37.095 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:37.095 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:37.100 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:37.100 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:37.123 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:37.141 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:37.144 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:37.247 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:37.285 INFO:tasks.workunit.client.0.smithi132.stdout: 49439 ? 00:00:00 ps 2024-03-27T06:18:37.286 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:37.301 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:37.305 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:37.314 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:37.314 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:37.319 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:37.319 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:37.342 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:37.359 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:37.361 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:37.431 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:37.467 INFO:tasks.workunit.client.0.smithi132.stdout: 49449 ? 00:00:00 ps 2024-03-27T06:18:37.468 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:37.487 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:37.493 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:37.505 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:37.505 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:37.511 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:37.511 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:37.532 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:37.550 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:37.553 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:37.670 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:37.706 INFO:tasks.workunit.client.0.smithi132.stdout: 49459 ? 00:00:00 ps 2024-03-27T06:18:37.707 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:37.726 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:37.730 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:37.740 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:37.740 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:37.746 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:37.746 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:37.768 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:37.785 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:37.788 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:37.869 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:37.902 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stdout: 49469 ? 00:00:00 ps 2024-03-27T06:18:37.903 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:37.923 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:37.927 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:37.938 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:37.938 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:37.942 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:37.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:37.966 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:37.983 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:37.986 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:38.082 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:38.118 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:38.118 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:38.118 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:38.118 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:38.118 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:38.119 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:38.119 INFO:tasks.workunit.client.0.smithi132.stdout: 49479 ? 00:00:00 ps 2024-03-27T06:18:38.119 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:38.141 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:38.147 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:38.159 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 20` 2024-03-27T06:18:38.159 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a 2024-03-27T06:18:38.164 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/one 2024-03-27T06:18:38.164 INFO:tasks.workunit.client.0.smithi132.stderr:+ cat 2024-03-27T06:18:38.186 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir a/.snap/two 2024-03-27T06:18:38.204 INFO:tasks.workunit.client.0.smithi132.stderr:+ chmod 777 a/foo 2024-03-27T06:18:38.206 INFO:tasks.workunit.client.0.smithi132.stderr:+ sync 2024-03-27T06:18:38.236 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:38.265 INFO:tasks.workunit.client.0.smithi132.stderr:+ ps 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: PID TTY TIME CMD 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 4289 ? 00:00:00 systemd 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 4291 ? 00:00:00 (sd-pam) 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 22966 ? 00:00:02 sshd 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 48425 ? 00:00:00 timeout 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 49186 ? 00:00:00 snaptest-double 2024-03-27T06:18:38.296 INFO:tasks.workunit.client.0.smithi132.stdout: 49503 ? 00:00:00 ps 2024-03-27T06:18:38.297 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-27T06:18:38.311 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm a/foo 2024-03-27T06:18:38.322 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir a 2024-03-27T06:18:38.342 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:18:38.343 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:18:38.344 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:18:38.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:18:38.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.453+0000 7f89eec00640 1 Processor -- start 2024-03-27T06:18:38.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.453+0000 7f89eec00640 1 -- start start 2024-03-27T06:18:38.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89eec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e8157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89eec00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f89e8073880 0x7f89e8157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89eec00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e806b520 con 0x7f89e81516f0 2024-03-27T06:18:38.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89eec00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f89e8072be0 con 0x7f89e8073880 2024-03-27T06:18:38.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89eec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f89e806b7c0 con 0x7f89e8151ad0 2024-03-27T06:18:38.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89e7e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f89e81516f0 0x7f89e80c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:36464/0 (socket says 172.21.15.132:36464) 2024-03-27T06:18:38.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89e7e00640 1 -- 172.21.15.132:0/3012526307 learned_addr learned my addr 172.21.15.132:0/3012526307 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:38.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89e7400640 1 --2- 172.21.15.132:0/3012526307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e8157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.454+0000 7f89ecc00640 1 --2- 172.21.15.132:0/3012526307 >> v2:172.21.15.132:3300/0 conn(0x7f89e8073880 0x7f89e8157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e6a00640 1 -- 172.21.15.132:0/3012526307 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528192634 0 0) 0x7f89e806b520 con 0x7f89e81516f0 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e6a00640 1 -- 172.21.15.132:0/3012526307 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89cc003520 con 0x7f89e81516f0 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e7400640 1 -- 172.21.15.132:0/3012526307 >> v2:172.21.15.132:3300/0 conn(0x7f89e8073880 msgr2=0x7f89e8157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e7400640 1 --2- 172.21.15.132:0/3012526307 >> v2:172.21.15.132:3300/0 conn(0x7f89e8073880 0x7f89e8157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e7400640 1 -- 172.21.15.132:0/3012526307 >> v1:172.21.15.134:6789/0 conn(0x7f89e81516f0 legacy=0x7f89e80c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.455+0000 7f89e7400640 1 -- 172.21.15.132:0/3012526307 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e806baa0 con 0x7f89e8151ad0 2024-03-27T06:18:38.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.457+0000 7f89e7400640 1 --2- 172.21.15.132:0/3012526307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e8157570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89d40097a0 tx=0x7f89d402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3383ea2764f49e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:38.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.457+0000 7f89e6a00640 1 -- 172.21.15.132:0/3012526307 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4034070 con 0x7f89e8151ad0 2024-03-27T06:18:38.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.457+0000 7f89e6a00640 1 -- 172.21.15.132:0/3012526307 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4030800 con 0x7f89e8151ad0 2024-03-27T06:18:38.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.457+0000 7f89e6a00640 1 -- 172.21.15.132:0/3012526307 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4030bd0 con 0x7f89e8151ad0 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 -- 172.21.15.132:0/3012526307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 msgr2=0x7f89e8157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 --2- 172.21.15.132:0/3012526307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e8157570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89d40097a0 tx=0x7f89d402b940 comp rx=0 tx=0).stop 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 -- 172.21.15.132:0/3012526307 shutdown_connections 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 --2- 172.21.15.132:0/3012526307 >> v2:172.21.15.132:3300/0 conn(0x7f89e8073880 0x7f89e8157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 --2- 172.21.15.132:0/3012526307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e8157570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 -- 172.21.15.132:0/3012526307 >> 172.21.15.132:0/3012526307 conn(0x7f89e80cc060 msgr2=0x7f89e80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 -- 172.21.15.132:0/3012526307 shutdown_connections 2024-03-27T06:18:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.458+0000 7f89eec00640 1 -- 172.21.15.132:0/3012526307 wait complete. 2024-03-27T06:18:38.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.460+0000 7f89eec00640 1 Processor -- start 2024-03-27T06:18:38.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.460+0000 7f89eec00640 1 -- start start 2024-03-27T06:18:38.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89e8073880 0x7f89e813dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e813f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89e806bd80 con 0x7f89e8073880 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8072870 con 0x7f89e81516f0 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89eec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f89e80c1080 con 0x7f89e8151ad0 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89e8073880 0x7f89e813dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:42314/0 (socket says 172.21.15.132:42314) 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 -- 172.21.15.132:0/2718060094 learned_addr learned my addr 172.21.15.132:0/2718060094 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89ecc00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e813f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 -- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 msgr2=0x7f89e813f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e813f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 -- 172.21.15.132:0/2718060094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89e8073880 msgr2=0x7f89e813dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89e8073880 0x7f89e813dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.461+0000 7f89e7400640 1 -- 172.21.15.132:0/2718060094 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e81f3af0 con 0x7f89e81516f0 2024-03-27T06:18:38.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89ecc00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e813f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:38.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89e7400640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009890 tx=0x7f89d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4034040 con 0x7f89e81516f0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4030e30 con 0x7f89e81516f0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40443f0 con 0x7f89e81516f0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e81f3dd0 con 0x7f89e81516f0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.462+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e81f4390 con 0x7f89e81516f0 2024-03-27T06:18:38.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.463+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e8068840 con 0x7f89e81516f0 2024-03-27T06:18:38.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.465+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f89d404b060 con 0x7f89e81516f0 2024-03-27T06:18:38.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.466+0000 7f89dfe00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 0x7f89c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:38.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.466+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5996+0+0 (secure 0 0 0) 0x7f89d40ad520 con 0x7f89e81516f0 2024-03-27T06:18:38.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.466+0000 7f89e7e00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 0x7f89c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:38.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.467+0000 7f89e7e00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 0x7f89c806e9f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89d8005880 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:38.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.470+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f89d403f070 con 0x7f89e81516f0 2024-03-27T06:18:38.477 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-03-27T06:18:38.477 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-27T06:18:38.584 INFO:tasks.workunit.client.0.smithi132.stderr:+ mkdir .snap/foo 2024-03-27T06:18:38.602 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-03-27T06:18:38.602 INFO:tasks.workunit.client.0.smithi132.stdout:We want ENOENT, not ESTALE, here. 2024-03-27T06:18:38.603 INFO:tasks.workunit.client.0.smithi132.stderr:++ seq 1 100 2024-03-27T06:18:38.605 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.605 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/1 2024-03-27T06:18:38.606 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.610 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-03-27T06:18:38.610 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.611 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/2 2024-03-27T06:18:38.611 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.614 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-03-27T06:18:38.615 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.615 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/3 2024-03-27T06:18:38.616 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.618 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-03-27T06:18:38.619 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.619 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/4 2024-03-27T06:18:38.620 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.622 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-03-27T06:18:38.622 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.623 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/5 2024-03-27T06:18:38.624 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.626 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-03-27T06:18:38.626 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.627 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/6 2024-03-27T06:18:38.627 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.630 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-03-27T06:18:38.630 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.631 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/7 2024-03-27T06:18:38.631 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.633 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-03-27T06:18:38.633 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.634 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/8 2024-03-27T06:18:38.634 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.637 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-03-27T06:18:38.637 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.638 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/9 2024-03-27T06:18:38.639 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.641 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-03-27T06:18:38.641 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.642 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/10 2024-03-27T06:18:38.642 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.644 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-03-27T06:18:38.645 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.645 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/11 2024-03-27T06:18:38.645 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.647 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-03-27T06:18:38.648 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.648 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/12 2024-03-27T06:18:38.649 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.651 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-03-27T06:18:38.651 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.651 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/13 2024-03-27T06:18:38.652 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.654 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-03-27T06:18:38.654 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.655 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/14 2024-03-27T06:18:38.655 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.657 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-03-27T06:18:38.658 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.658 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/15 2024-03-27T06:18:38.659 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.661 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-03-27T06:18:38.661 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.662 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/16 2024-03-27T06:18:38.662 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.665 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-03-27T06:18:38.666 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/17 2024-03-27T06:18:38.667 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.670 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-03-27T06:18:38.671 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.672 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/18 2024-03-27T06:18:38.672 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.676 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-03-27T06:18:38.676 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.677 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/19 2024-03-27T06:18:38.677 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.679 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-03-27T06:18:38.680 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.680 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/20 2024-03-27T06:18:38.680 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.683 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-03-27T06:18:38.683 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.684 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/21 2024-03-27T06:18:38.684 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.686 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-03-27T06:18:38.690 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.690 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/22 2024-03-27T06:18:38.690 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.690 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-03-27T06:18:38.691 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.691 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/23 2024-03-27T06:18:38.691 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.695 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-03-27T06:18:38.695 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.696 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/24 2024-03-27T06:18:38.696 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.699 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-03-27T06:18:38.700 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.700 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/25 2024-03-27T06:18:38.701 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.703 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-03-27T06:18:38.703 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.704 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/26 2024-03-27T06:18:38.704 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.707 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-03-27T06:18:38.707 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.708 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/27 2024-03-27T06:18:38.709 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.710 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-03-27T06:18:38.711 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.711 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/28 2024-03-27T06:18:38.712 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.715 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-03-27T06:18:38.715 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.716 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/29 2024-03-27T06:18:38.716 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.719 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-03-27T06:18:38.720 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.721 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/30 2024-03-27T06:18:38.721 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.723 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-03-27T06:18:38.724 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.724 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/31 2024-03-27T06:18:38.725 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.728 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-03-27T06:18:38.728 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.729 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/32 2024-03-27T06:18:38.730 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.733 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-03-27T06:18:38.733 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.734 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/33 2024-03-27T06:18:38.735 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.737 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-03-27T06:18:38.737 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.738 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/34 2024-03-27T06:18:38.739 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.740 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-03-27T06:18:38.741 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.741 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/35 2024-03-27T06:18:38.742 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.744 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-03-27T06:18:38.745 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.746 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/36 2024-03-27T06:18:38.746 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.748 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-03-27T06:18:38.749 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.750 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/37 2024-03-27T06:18:38.750 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.752 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-03-27T06:18:38.753 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.753 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/38 2024-03-27T06:18:38.754 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.756 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-03-27T06:18:38.756 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.758 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/39 2024-03-27T06:18:38.758 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.760 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-03-27T06:18:38.760 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.761 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/40 2024-03-27T06:18:38.761 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.764 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-03-27T06:18:38.764 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.765 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/41 2024-03-27T06:18:38.765 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.767 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-03-27T06:18:38.768 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.769 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/42 2024-03-27T06:18:38.769 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.772 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-03-27T06:18:38.773 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.773 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/43 2024-03-27T06:18:38.774 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.777 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-03-27T06:18:38.778 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.778 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/44 2024-03-27T06:18:38.779 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.781 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-03-27T06:18:38.781 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.782 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/45 2024-03-27T06:18:38.782 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.784 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-03-27T06:18:38.785 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.785 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/46 2024-03-27T06:18:38.786 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.798+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89e81481f0 con 0x7f89e81516f0 2024-03-27T06:18:38.788 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-03-27T06:18:38.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.799+0000 7f89dfe00640 1 -- 172.21.15.132:0/2718060094 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7f89d407b460 con 0x7f89e81516f0 2024-03-27T06:18:38.789 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:38.789 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:18:38.789 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:18:38.790 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.790 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/47 2024-03-27T06:18:38.790 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 msgr2=0x7f89c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 0x7f89c806e9f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89d8005880 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 msgr2=0x7f89e8154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009890 tx=0x7f89d40046c0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 shutdown_connections 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f89e8151ad0 0x7f89e813f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f89c806c530 0x7f89c806e9f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89e81516f0 0x7f89e8154170 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 --2- 172.21.15.132:0/2718060094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89e8073880 0x7f89e813dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 >> 172.21.15.132:0/2718060094 conn(0x7f89e80cc060 msgr2=0x7f89e806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:38.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 shutdown_connections 2024-03-27T06:18:38.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:38.801+0000 7f89eec00640 1 -- 172.21.15.132:0/2718060094 wait complete. 2024-03-27T06:18:38.793 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-03-27T06:18:38.793 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.794 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/48 2024-03-27T06:18:38.795 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.797 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-03-27T06:18:38.797 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.798 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/49 2024-03-27T06:18:38.798 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.802 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-03-27T06:18:38.802 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.803 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/50 2024-03-27T06:18:38.803 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.806 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-03-27T06:18:38.807 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.808 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/51 2024-03-27T06:18:38.808 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.810 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-03-27T06:18:38.810 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.811 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/52 2024-03-27T06:18:38.811 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.814 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-03-27T06:18:38.815 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.816 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-03-27T06:18:38.816 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2024-03-27T06:18:38.817 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/53 2024-03-27T06:18:38.817 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.818 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-03-27T06:18:38.819 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.819 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/54 2024-03-27T06:18:38.820 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.822 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-03-27T06:18:38.822 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.823 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/55 2024-03-27T06:18:38.823 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.825 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-03-27T06:18:38.826 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.827 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/56 2024-03-27T06:18:38.827 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.828 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-03-27T06:18:38.829 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.829 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/57 2024-03-27T06:18:38.830 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.832 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-03-27T06:18:38.832 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.833 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/58 2024-03-27T06:18:38.833 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.835 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-03-27T06:18:38.836 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.836 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/59 2024-03-27T06:18:38.837 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.838 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-03-27T06:18:38.839 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.839 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/60 2024-03-27T06:18:38.840 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.842 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-03-27T06:18:38.842 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.843 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/61 2024-03-27T06:18:38.843 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.845 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-03-27T06:18:38.845 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.846 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/62 2024-03-27T06:18:38.846 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.848 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-03-27T06:18:38.849 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.849 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.850 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/63 2024-03-27T06:18:38.852 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-03-27T06:18:38.852 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.853 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/64 2024-03-27T06:18:38.853 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.855 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-03-27T06:18:38.856 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.856 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/65 2024-03-27T06:18:38.857 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.859 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-03-27T06:18:38.859 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.860 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/66 2024-03-27T06:18:38.860 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.862 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-03-27T06:18:38.863 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.863 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/67 2024-03-27T06:18:38.864 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.865 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-03-27T06:18:38.866 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.867 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.867 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/68 2024-03-27T06:18:38.869 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-03-27T06:18:38.869 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.870 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/69 2024-03-27T06:18:38.870 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.872 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-03-27T06:18:38.873 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.873 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/70 2024-03-27T06:18:38.874 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.876 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-03-27T06:18:38.876 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.877 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/71 2024-03-27T06:18:38.877 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.879 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-03-27T06:18:38.879 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.880 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/72 2024-03-27T06:18:38.880 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.883 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-03-27T06:18:38.883 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.883 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/73 2024-03-27T06:18:38.884 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.885 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-03-27T06:18:38.886 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.886 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/74 2024-03-27T06:18:38.887 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.890 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-03-27T06:18:38.893 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.894 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/75 2024-03-27T06:18:38.894 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.894 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-03-27T06:18:38.895 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.895 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/76 2024-03-27T06:18:38.895 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.897 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-03-27T06:18:38.898 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.898 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/77 2024-03-27T06:18:38.899 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.901 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-03-27T06:18:38.901 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.902 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/78 2024-03-27T06:18:38.903 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.905 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-03-27T06:18:38.905 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.906 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/79 2024-03-27T06:18:38.907 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.909 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-03-27T06:18:38.910 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.910 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/80 2024-03-27T06:18:38.911 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.915 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-03-27T06:18:38.916 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.917 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/81 2024-03-27T06:18:38.917 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.919 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-03-27T06:18:38.920 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.920 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/82 2024-03-27T06:18:38.921 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.923 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-03-27T06:18:38.923 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.923 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/83 2024-03-27T06:18:38.924 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.926 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-03-27T06:18:38.926 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.927 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/84 2024-03-27T06:18:38.927 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.929 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-03-27T06:18:38.929 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.930 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/85 2024-03-27T06:18:38.930 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.933 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-03-27T06:18:38.933 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.933 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/86 2024-03-27T06:18:38.934 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.936 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-03-27T06:18:38.936 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.936 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/87 2024-03-27T06:18:38.937 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.939 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-03-27T06:18:38.939 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.940 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/88 2024-03-27T06:18:38.940 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.942 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-03-27T06:18:38.942 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/89 2024-03-27T06:18:38.943 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.945 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-03-27T06:18:38.946 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.946 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/90 2024-03-27T06:18:38.946 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.949 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-03-27T06:18:38.949 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.949 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/91 2024-03-27T06:18:38.950 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.952 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-03-27T06:18:38.952 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.953 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/92 2024-03-27T06:18:38.953 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.955 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-03-27T06:18:38.956 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.956 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/93 2024-03-27T06:18:38.956 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.958 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-03-27T06:18:38.959 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.959 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/94 2024-03-27T06:18:38.960 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.962 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-03-27T06:18:38.962 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.963 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/95 2024-03-27T06:18:38.963 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.965 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-03-27T06:18:38.965 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.966 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/96 2024-03-27T06:18:38.966 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.968 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-03-27T06:18:38.969 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.969 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/97 2024-03-27T06:18:38.970 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.972 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-03-27T06:18:38.972 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.973 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/98 2024-03-27T06:18:38.973 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.975 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-03-27T06:18:38.975 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.976 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/99 2024-03-27T06:18:38.976 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.979 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-03-27T06:18:38.979 INFO:tasks.workunit.client.0.smithi132.stderr:+ for f in `seq 1 100` 2024-03-27T06:18:38.980 INFO:tasks.workunit.client.0.smithi132.stderr:+ stat .snap/foo/100 2024-03-27T06:18:38.982 INFO:tasks.workunit.client.0.smithi132.stderr:+ grep 'No such file' 2024-03-27T06:18:38.983 INFO:tasks.workunit.client.0.smithi132.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-03-27T06:18:38.983 INFO:tasks.workunit.client.0.smithi132.stderr:+ rmdir .snap/foo 2024-03-27T06:18:38.992 INFO:tasks.workunit.client.0.smithi132.stdout:OK 2024-03-27T06:18:38.993 INFO:tasks.workunit.client.0.smithi132.stderr:+ echo OK 2024-03-27T06:18:38.993 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-27T06:18:38.993 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-27T06:18:39.113 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-03-27T06:18:39.115 DEBUG:teuthology.orchestra.run.smithi132:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-27T06:18:39.167 INFO:tasks.workunit.client.0.smithi132.stderr:+ set -e 2024-03-27T06:18:39.168 INFO:tasks.workunit.client.0.smithi132.stderr:+ sudo git config --global http.sslVerify false 2024-03-27T06:18:39.209 INFO:tasks.workunit.client.0.smithi132.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-03-27T06:18:39.256 INFO:tasks.workunit.client.0.smithi132.stderr:+ retried=false 2024-03-27T06:18:39.256 INFO:tasks.workunit.client.0.smithi132.stderr:+ trap -- retry EXIT 2024-03-27T06:18:39.256 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm -rf ceph 2024-03-27T06:18:39.259 INFO:tasks.workunit.client.0.smithi132.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-03-27T06:18:39.266 INFO:tasks.workunit.client.0.smithi132.stderr:Cloning into 'ceph'... 2024-03-27T06:18:55.818 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:56.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.071+0000 7ff6d5e00640 1 Processor -- start 2024-03-27T06:18:56.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.071+0000 7ff6d5e00640 1 -- start start 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 0x7ff6d00727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 0x7ff6d0158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d01594f0 con 0x7ff6d006b520 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5e00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d0152eb0 con 0x7ff6d0158b60 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00c7bc0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d5400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 0x7ff6d0158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6d4a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 0x7ff6d00727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6c7e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ff6d006b520 0x7ff6d00c62f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:50328/0 (socket says 172.21.15.132:50328) 2024-03-27T06:18:56.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.073+0000 7ff6c7e00640 1 -- 172.21.15.132:0/1749048986 learned_addr learned my addr 172.21.15.132:0/1749048986 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:56.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.074+0000 7ff6d4a00640 1 -- 172.21.15.132:0/1749048986 >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 msgr2=0x7ff6d0158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:56.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.074+0000 7ff6d4a00640 1 --2- 172.21.15.132:0/1749048986 >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 0x7ff6d0158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.074+0000 7ff6d4a00640 1 -- 172.21.15.132:0/1749048986 >> v1:172.21.15.132:6789/0 conn(0x7ff6d006b520 legacy=0x7ff6d00c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.074+0000 7ff6d4a00640 1 -- 172.21.15.132:0/1749048986 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00c7ea0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.074+0000 7ff6d5400640 1 --2- 172.21.15.132:0/1749048986 >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 0x7ff6d0158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:56.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.075+0000 7ff6d4a00640 1 --2- 172.21.15.132:0/1749048986 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 0x7ff6d00727d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00097a0 tx=0x7ff6c002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e3e230eea02900 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.075+0000 7ff6cfe00640 1 -- 172.21.15.132:0/1749048986 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0031780 con 0x7ff6d0152ad0 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.075+0000 7ff6cfe00640 1 -- 172.21.15.132:0/1749048986 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c0037070 con 0x7ff6d0152ad0 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.075+0000 7ff6cfe00640 1 -- 172.21.15.132:0/1749048986 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0031b40 con 0x7ff6d0152ad0 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 -- 172.21.15.132:0/1749048986 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 msgr2=0x7ff6d00727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/1749048986 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 0x7ff6d00727d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00097a0 tx=0x7ff6c002b820 comp rx=0 tx=0).stop 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 -- 172.21.15.132:0/1749048986 shutdown_connections 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/1749048986 >> v2:172.21.15.134:3300/0 conn(0x7ff6d0158b60 0x7ff6d0158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/1749048986 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0152ad0 0x7ff6d00727d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 -- 172.21.15.132:0/1749048986 >> 172.21.15.132:0/1749048986 conn(0x7ff6d006fae0 msgr2=0x7ff6d006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:56.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 -- 172.21.15.132:0/1749048986 shutdown_connections 2024-03-27T06:18:56.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.076+0000 7ff6d5e00640 1 -- 172.21.15.132:0/1749048986 wait complete. 2024-03-27T06:18:56.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.079+0000 7ff6d5e00640 1 Processor -- start 2024-03-27T06:18:56.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.079+0000 7ff6d5e00640 1 -- start start 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.080+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 0x7ff6d01f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.080+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.080+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 0x7ff6d014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d5e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0072d10 con 0x7ff6d006b520 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d5e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d00b2d70 con 0x7ff6d0152ad0 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d5e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00c76a0 con 0x7ff6d0158b60 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d4a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 0x7ff6d01f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d5400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 0x7ff6d014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d4a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 0x7ff6d01f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51984/0 (socket says 172.21.15.132:51984) 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57318/0 (socket says 172.21.15.132:57318) 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d4a00640 1 -- 172.21.15.132:0/193736347 learned_addr learned my addr 172.21.15.132:0/193736347 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 -- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 msgr2=0x7ff6d014bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 0x7ff6d014bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 -- 172.21.15.132:0/193736347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 msgr2=0x7ff6d01f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 0x7ff6d01f18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6c7e00640 1 -- 172.21.15.132:0/193736347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d01de820 con 0x7ff6d0152ad0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.081+0000 7ff6d5400640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 0x7ff6d014bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6c7e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000f60 tx=0x7ff6b800c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8017070 con 0x7ff6d0152ad0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b8005ce0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8007310 con 0x7ff6d0152ad0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d01558e0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.082+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d0157af0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.083+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d00758a0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.084+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff6b801a070 con 0x7ff6d0152ad0 2024-03-27T06:18:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.085+0000 7ff6cd600640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 0x7ff6ac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.085+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7ff6b808a970 con 0x7ff6d0152ad0 2024-03-27T06:18:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.085+0000 7ff6d4a00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 0x7ff6ac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.086+0000 7ff6d4a00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 0x7ff6ac06e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff6c002b860 tx=0x7ff6c003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.089+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff6b8058800 con 0x7ff6d0152ad0 2024-03-27T06:18:56.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.330+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6d00733a0 con 0x7ff6d0152ad0 2024-03-27T06:18:56.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.331+0000 7ff6cd600640 1 -- 172.21.15.132:0/193736347 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7ff6b805cb00 con 0x7ff6d0152ad0 2024-03-27T06:18:56.321 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:56.321 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:18:56.322 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.334+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 msgr2=0x7ff6ac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.334+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 0x7ff6ac06e9f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff6c002b860 tx=0x7ff6c003d040 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 msgr2=0x7ff6d00cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000f60 tx=0x7ff6b800c760 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 shutdown_connections 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff6d0158b60 0x7ff6d014bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff6ac06c530 0x7ff6ac06e9f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6d0152ad0 0x7ff6d00cb8b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 --2- 172.21.15.132:0/193736347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6d006b520 0x7ff6d01f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 >> 172.21.15.132:0/193736347 conn(0x7ff6d006fae0 msgr2=0x7ff6d01509a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 shutdown_connections 2024-03-27T06:18:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.335+0000 7ff6d5e00640 1 -- 172.21.15.132:0/193736347 wait complete. 2024-03-27T06:18:56.350 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-03-27T06:18:56.351 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-27T06:18:56.351 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:18:56.362 INFO:tasks.ceph.mds.f:Stopped 2024-03-27T06:18:56.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:56.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.559+0000 7f860b600640 1 Processor -- start 2024-03-27T06:18:56.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.559+0000 7f860b600640 1 -- start start 2024-03-27T06:18:56.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f860b600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8604152950 0x7f86041590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f860b600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 0x7f8604071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f860b600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860406b630 con 0x7f8604073970 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f860b600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f8604152d30 con 0x7f8604152950 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f860b600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f860406b8d0 con 0x7f8604073550 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f8608c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8604152950 0x7f86041590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 0x7f8604071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f8609600640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8604073970 0x7f860406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:50342/0 (socket says 172.21.15.132:50342) 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.561+0000 7f8609600640 1 -- 172.21.15.132:0/2837370621 learned_addr learned my addr 172.21.15.132:0/2837370621 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.562+0000 7f8608c00640 1 -- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 msgr2=0x7f8604071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.562+0000 7f8608c00640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 0x7f8604071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.562+0000 7f8608c00640 1 -- 172.21.15.132:0/2837370621 >> v1:172.21.15.132:6789/0 conn(0x7f8604073970 legacy=0x7f860406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.562+0000 7f8608c00640 1 -- 172.21.15.132:0/2837370621 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860406bbb0 con 0x7f8604152950 2024-03-27T06:18:56.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.562+0000 7f85fbe00640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 0x7f8604071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:56.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.563+0000 7f8608c00640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604152950 0x7f86041590a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f85f40097a0 tx=0x7f85f402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f5c835fe856db server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.563+0000 7f8603e00640 1 -- 172.21.15.132:0/2837370621 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4035020 con 0x7f8604152950 2024-03-27T06:18:56.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.563+0000 7f8603e00640 1 -- 172.21.15.132:0/2837370621 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f4031810 con 0x7f8604152950 2024-03-27T06:18:56.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.563+0000 7f8603e00640 1 -- 172.21.15.132:0/2837370621 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4031be0 con 0x7f8604152950 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 -- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604152950 msgr2=0x7f86041590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604152950 0x7f86041590a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f85f40097a0 tx=0x7f85f402b8b0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 -- 172.21.15.132:0/2837370621 shutdown_connections 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604073550 0x7f8604071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 --2- 172.21.15.132:0/2837370621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604152950 0x7f86041590a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 -- 172.21.15.132:0/2837370621 >> 172.21.15.132:0/2837370621 conn(0x7f86040cc060 msgr2=0x7f86040cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 -- 172.21.15.132:0/2837370621 shutdown_connections 2024-03-27T06:18:56.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.564+0000 7f860b600640 1 -- 172.21.15.132:0/2837370621 wait complete. 2024-03-27T06:18:56.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.565+0000 7f860b600640 1 Processor -- start 2024-03-27T06:18:56.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 -- start start 2024-03-27T06:18:56.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 0x7f86041f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604152950 0x7f86041577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f860406c450 con 0x7f8604073970 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.566+0000 7f860b600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f860406c170 con 0x7f8604073550 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f860b600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f860406d660 con 0x7f8604152950 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51992/0 (socket says 172.21.15.132:51992) 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f85fbe00640 1 -- 172.21.15.132:0/871260732 learned_addr learned my addr 172.21.15.132:0/871260732 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8609600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604152950 0x7f86041577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 0x7f86041f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 -- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604152950 msgr2=0x7f86041577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604152950 0x7f86041577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 -- 172.21.15.132:0/871260732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 msgr2=0x7f86041f0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f8608c00640 1 -- 172.21.15.132:0/871260732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86041585f0 con 0x7f8604073550 2024-03-27T06:18:56.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.567+0000 7f85fbe00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:56.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f8608c00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 0x7f86041f4140 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8604070b50 tx=0x7f85f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4035040 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f4039ae0 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4042420 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86041588d0 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.568+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86041f05e0 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.570+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8604068840 con 0x7f8604073550 2024-03-27T06:18:56.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.570+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f85f4038020 con 0x7f8604073550 2024-03-27T06:18:56.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.571+0000 7f8601600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 0x7f85e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:56.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.571+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7f85f40acf60 con 0x7f8604073550 2024-03-27T06:18:56.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.571+0000 7f85fbe00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 0x7f85e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:56.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.572+0000 7f85fbe00640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 0x7f85e006e9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f86040ca4f0 tx=0x7f85ec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:56.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.575+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f85f407ad40 con 0x7f8604073550 2024-03-27T06:18:56.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.936+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86040c6b30 con 0x7f8604073550 2024-03-27T06:18:56.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.937+0000 7f8601600640 1 -- 172.21.15.132:0/871260732 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7f85f407f040 con 0x7f8604073550 2024-03-27T06:18:56.927 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:56.927 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:18:56.927 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:18:56.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 msgr2=0x7f85e006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 0x7f85e006e9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f86040ca4f0 tx=0x7f85ec009190 comp rx=0 tx=0).stop 2024-03-27T06:18:56.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 msgr2=0x7f86041f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 0x7f86041f4140 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8604070b50 tx=0x7f85f40046c0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 shutdown_connections 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8604152950 0x7f86041577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8604073970 0x7f86041f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f85e006c530 0x7f85e006e9f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 --2- 172.21.15.132:0/871260732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8604073550 0x7f86041f4140 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.939+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 >> 172.21.15.132:0/871260732 conn(0x7f86040cc060 msgr2=0x7f860406da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.940+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 shutdown_connections 2024-03-27T06:18:56.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:56.940+0000 7f860b600640 1 -- 172.21.15.132:0/871260732 wait complete. 2024-03-27T06:18:56.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-27T06:18:58.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:18:59.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.168+0000 7fd6fd400640 1 Processor -- start 2024-03-27T06:18:59.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.168+0000 7fd6fd400640 1 -- start start 2024-03-27T06:18:59.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd6f806b790 0x7fd6f8168030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd6f806bb70 0x7fd6f808c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 0x7fd6f808c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f8168570 con 0x7fd6f806b790 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f80c1200 con 0x7fd6f806bb70 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.169+0000 7fd6fd400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f80c14a0 con 0x7fd6f806bf90 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.170+0000 7fd6f6800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd6f806b790 0x7fd6f8168030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.170+0000 7fd6f6800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd6f806b790 0x7fd6f8168030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51998/0 (socket says 172.21.15.132:51998) 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.170+0000 7fd6f5e00640 1 --2- 172.21.15.132:0/1426356293 >> v2:172.21.15.134:3300/0 conn(0x7fd6f806bb70 0x7fd6f808c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.170+0000 7fd6f7200640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 0x7fd6f808c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.170+0000 7fd6f6800640 1 -- 172.21.15.132:0/1426356293 learned_addr learned my addr 172.21.15.132:0/1426356293 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f6800640 1 -- 172.21.15.132:0/1426356293 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 msgr2=0x7fd6f808c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f6800640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 0x7fd6f808c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f6800640 1 -- 172.21.15.132:0/1426356293 >> v2:172.21.15.134:3300/0 conn(0x7fd6f806bb70 msgr2=0x7fd6f808c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:18:59.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f6800640 1 --2- 172.21.15.132:0/1426356293 >> v2:172.21.15.134:3300/0 conn(0x7fd6f806bb70 0x7fd6f808c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f6800640 1 -- 172.21.15.132:0/1426356293 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80c1780 con 0x7fd6f806b790 2024-03-27T06:18:59.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.226+0000 7fd6f7200640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 0x7fd6f808c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:18:59.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.228+0000 7fd6f6800640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806b790 0x7fd6f8168030 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80c6720 tx=0x7fd6e8034910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ddc1b0423a153c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:59.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.228+0000 7fd6f5400640 1 -- 172.21.15.132:0/1426356293 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e803b780 con 0x7fd6f806b790 2024-03-27T06:18:59.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.228+0000 7fd6f5400640 1 -- 172.21.15.132:0/1426356293 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e8041070 con 0x7fd6f806b790 2024-03-27T06:18:59.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.228+0000 7fd6f5400640 1 -- 172.21.15.132:0/1426356293 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e803bb40 con 0x7fd6f806b790 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 -- 172.21.15.132:0/1426356293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806b790 msgr2=0x7fd6f8168030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806b790 0x7fd6f8168030 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80c6720 tx=0x7fd6e8034910 comp rx=0 tx=0).stop 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 -- 172.21.15.132:0/1426356293 shutdown_connections 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bf90 0x7fd6f808c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 --2- 172.21.15.132:0/1426356293 >> v2:172.21.15.134:3300/0 conn(0x7fd6f806bb70 0x7fd6f808c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 --2- 172.21.15.132:0/1426356293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806b790 0x7fd6f8168030 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 -- 172.21.15.132:0/1426356293 >> 172.21.15.132:0/1426356293 conn(0x7fd6f80cc060 msgr2=0x7fd6f80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:59.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 -- 172.21.15.132:0/1426356293 shutdown_connections 2024-03-27T06:18:59.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.230+0000 7fd6fd400640 1 -- 172.21.15.132:0/1426356293 wait complete. 2024-03-27T06:18:59.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.232+0000 7fd6fd400640 1 Processor -- start 2024-03-27T06:18:59.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.232+0000 7fd6fd400640 1 -- start start 2024-03-27T06:18:59.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6f806b790 0x7fd6f80c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bb70 0x7fd6f8200c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f80739c0 con 0x7fd6f806bf90 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f80c1d40 con 0x7fd6f806b790 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6fd400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f8073b40 con 0x7fd6f806bb70 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6f7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6f7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52010/0 (socket says 172.21.15.132:52010) 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6f7200640 1 -- 172.21.15.132:0/102952436 learned_addr learned my addr 172.21.15.132:0/102952436 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6f5e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bb70 0x7fd6f8200c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.233+0000 7fd6f6800640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6f806b790 0x7fd6f80c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 -- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bb70 msgr2=0x7fd6f8200c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bb70 0x7fd6f8200c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 -- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6f806b790 msgr2=0x7fd6f80c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6f806b790 0x7fd6f80c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 -- 172.21.15.132:0/102952436 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f82011c0 con 0x7fd6f806bf90 2024-03-27T06:18:59.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6f7200640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000eae0 tx=0x7fd6f000c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:59.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f000ca40 con 0x7fd6f806bf90 2024-03-27T06:18:59.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f000cc00 con 0x7fd6f806bf90 2024-03-27T06:18:59.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6fd400640 1 -- 172.21.15.132:0/102952436 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f80c5ba0 con 0x7fd6f806bf90 2024-03-27T06:18:59.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0003990 con 0x7fd6f806bf90 2024-03-27T06:18:59.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.234+0000 7fd6fd400640 1 -- 172.21.15.132:0/102952436 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f80c6140 con 0x7fd6f806bf90 2024-03-27T06:18:59.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.236+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc005290 con 0x7fd6f806bf90 2024-03-27T06:18:59.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.237+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd6f001d030 con 0x7fd6f806bf90 2024-03-27T06:18:59.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.237+0000 7fd6eea00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 0x7fd6d806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:18:59.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.238+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7fd6f008b050 con 0x7fd6f806bf90 2024-03-27T06:18:59.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.238+0000 7fd6f6800640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 0x7fd6d806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:18:59.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.238+0000 7fd6f6800640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 0x7fd6d806e9f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8040000 tx=0x7fd6e8047040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:18:59.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.242+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd6f0024060 con 0x7fd6f806bf90 2024-03-27T06:18:59.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.653+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6cc005850 con 0x7fd6f806bf90 2024-03-27T06:18:59.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.654+0000 7fd6eea00640 1 -- 172.21.15.132:0/102952436 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7fd6f0058f50 con 0x7fd6f806bf90 2024-03-27T06:18:59.644 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:18:59.644 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:18:59.645 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:18:59.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 msgr2=0x7fd6d806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 0x7fd6d806e9f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8040000 tx=0x7fd6e8047040 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 msgr2=0x7fd6f8166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000eae0 tx=0x7fd6f000c0e0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 shutdown_connections 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f806bf90 0x7fd6f8166220 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6f806bb70 0x7fd6f8200c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd6d806c530 0x7fd6d806e9f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.656+0000 7fd6e3e00640 1 --2- 172.21.15.132:0/102952436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6f806b790 0x7fd6f80c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.657+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 >> 172.21.15.132:0/102952436 conn(0x7fd6f80cc060 msgr2=0x7fd6f80cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.657+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 shutdown_connections 2024-03-27T06:18:59.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:18:59.657+0000 7fd6e3e00640 1 -- 172.21.15.132:0/102952436 wait complete. 2024-03-27T06:18:59.663 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-27T06:19:01.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:01.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.886+0000 7f4aa1400640 1 Processor -- start 2024-03-27T06:19:01.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.886+0000 7f4aa1400640 1 -- start start 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa1400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 0x7f4a9c0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa1400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4a9c15adb0 0x7f4a9c15b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa1400640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9c15de10 con 0x7f4a9c0cb870 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa1400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4a9c0cada0 con 0x7f4a9c15adb0 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa1400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f4a9c15e0b0 con 0x7f4a9c1549a0 2024-03-27T06:19:01.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4aa0a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4a9c15adb0 0x7f4a9c15b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4a9b400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f4a9c0cb870 0x7f4a9c06b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33622/0 (socket says 172.21.15.132:33622) 2024-03-27T06:19:01.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4a9b400640 1 -- 172.21.15.132:0/3054395450 learned_addr learned my addr 172.21.15.132:0/3054395450 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:01.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.889+0000 7f4a9be00640 1 --2- 172.21.15.132:0/3054395450 >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 0x7f4a9c0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4aa0a00640 1 -- 172.21.15.132:0/3054395450 >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 msgr2=0x7f4a9c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:01.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4aa0a00640 1 --2- 172.21.15.132:0/3054395450 >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 0x7f4a9c0cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:01.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4aa0a00640 1 -- 172.21.15.132:0/3054395450 >> v1:172.21.15.132:6789/0 conn(0x7f4a9c0cb870 legacy=0x7f4a9c06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:01.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4aa0a00640 1 -- 172.21.15.132:0/3054395450 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c15e390 con 0x7f4a9c15adb0 2024-03-27T06:19:01.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4a9be00640 1 --2- 172.21.15.132:0/3054395450 >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 0x7f4a9c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:01.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.890+0000 7f4a9aa00640 1 -- 172.21.15.132:0/3054395450 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650213675 0 0) 0x7f4a9c15de10 con 0x7f4a9c0cb870 2024-03-27T06:19:01.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.891+0000 7f4aa0a00640 1 --2- 172.21.15.132:0/3054395450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c15b1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0c73e0 tx=0x7f4a9402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fef19c87564a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:01.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.891+0000 7f4a9aa00640 1 -- 172.21.15.132:0/3054395450 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94039070 con 0x7f4a9c15adb0 2024-03-27T06:19:01.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.891+0000 7f4a9aa00640 1 -- 172.21.15.132:0/3054395450 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a9402cd10 con 0x7f4a9c15adb0 2024-03-27T06:19:01.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.891+0000 7f4a9aa00640 1 -- 172.21.15.132:0/3054395450 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94034660 con 0x7f4a9c15adb0 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 -- 172.21.15.132:0/3054395450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 msgr2=0x7f4a9c15b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 --2- 172.21.15.132:0/3054395450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c15b1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0c73e0 tx=0x7f4a9402bf60 comp rx=0 tx=0).stop 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 -- 172.21.15.132:0/3054395450 shutdown_connections 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 --2- 172.21.15.132:0/3054395450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c15b1c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 --2- 172.21.15.132:0/3054395450 >> v2:172.21.15.134:3300/0 conn(0x7f4a9c1549a0 0x7f4a9c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 -- 172.21.15.132:0/3054395450 >> 172.21.15.132:0/3054395450 conn(0x7f4a9c06d8d0 msgr2=0x7f4a9c06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 -- 172.21.15.132:0/3054395450 shutdown_connections 2024-03-27T06:19:01.884 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.892+0000 7f4aa1400640 1 -- 172.21.15.132:0/3054395450 wait complete. 2024-03-27T06:19:01.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.894+0000 7f4aa1400640 1 Processor -- start 2024-03-27T06:19:01.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.894+0000 7f4aa1400640 1 -- start start 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4a9c0cb870 0x7f4a9c1f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a9c1549a0 0x7f4a9c14f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c15ec30 con 0x7f4a9c15adb0 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c15e950 con 0x7f4a9c1549a0 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa1400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9c15b700 con 0x7f4a9c0cb870 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51864/0 (socket says 172.21.15.132:51864) 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.895+0000 7f4aa0a00640 1 -- 172.21.15.132:0/1234076332 learned_addr learned my addr 172.21.15.132:0/1234076332 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4a9be00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4a9c0cb870 0x7f4a9c1f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4a9b400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a9c1549a0 0x7f4a9c14f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 -- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4a9c0cb870 msgr2=0x7f4a9c1f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4a9c0cb870 0x7f4a9c1f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 -- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a9c1549a0 msgr2=0x7f4a9c14f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a9c1549a0 0x7f4a9c14f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 -- 172.21.15.132:0/1234076332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c1f4690 con 0x7f4a9c15adb0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.896+0000 7f4aa0a00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009e80 tx=0x7f4a94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.897+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94039050 con 0x7f4a9c15adb0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.897+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a94045070 con 0x7f4a9c15adb0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.897+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c158b50 con 0x7f4a9c15adb0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.897+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94009450 con 0x7f4a9c15adb0 2024-03-27T06:19:01.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.897+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c15d680 con 0x7f4a9c15adb0 2024-03-27T06:19:01.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.898+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9c0cb690 con 0x7f4a9c15adb0 2024-03-27T06:19:01.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.899+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4a9403e850 con 0x7f4a9c15adb0 2024-03-27T06:19:01.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.900+0000 7f4a93e00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 0x7f4a7c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:01.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.900+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7f4a940ae6a0 con 0x7f4a9c15adb0 2024-03-27T06:19:01.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.900+0000 7f4a9be00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 0x7f4a7c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:01.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.901+0000 7f4a9be00640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 0x7f4a7c06e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009560 tx=0x7f4a8c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:01.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:01.905+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4a9c0cb690 con 0x7f4a9c15adb0 2024-03-27T06:19:02.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.140+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a9c0686c0 con 0x7f4a9c15adb0 2024-03-27T06:19:02.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.141+0000 7f4a93e00640 1 -- 172.21.15.132:0/1234076332 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7f4a9c158b50 con 0x7f4a9c15adb0 2024-03-27T06:19:02.132 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:02.132 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:19:02.132 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:19:02.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 msgr2=0x7f4a7c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 0x7f4a7c06e9f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009560 tx=0x7f4a8c009190 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 msgr2=0x7f4a9c0c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009e80 tx=0x7f4a94004a40 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 shutdown_connections 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c15adb0 0x7f4a9c0c5b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a7c06c530 0x7f4a7c06e9f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a9c1549a0 0x7f4a9c14f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 --2- 172.21.15.132:0/1234076332 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4a9c0cb870 0x7f4a9c1f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 >> 172.21.15.132:0/1234076332 conn(0x7f4a9c06d8d0 msgr2=0x7f4a9c149830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 shutdown_connections 2024-03-27T06:19:02.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:02.143+0000 7f4aa1400640 1 -- 172.21.15.132:0/1234076332 wait complete. 2024-03-27T06:19:02.151 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-27T06:19:04.152 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:04.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:19:04.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:19:04.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.390+0000 7fa49b800640 1 Processor -- start 2024-03-27T06:19:04.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.391+0000 7fa49b800640 1 -- start start 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa49b800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa4940d0970 0x7fa49415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa49b800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa4940d0d50 0x7fa494154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa49b800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4940cb9f0 con 0x7fa4940cb440 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa49b800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa49415e2e0 con 0x7fa4940d0d50 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa49b800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa49415e5c0 con 0x7fa4940d0970 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa493e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa4940d0d50 0x7fa494154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa493e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa4940d0d50 0x7fa494154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51878/0 (socket says 172.21.15.132:51878) 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa493e00640 1 -- 172.21.15.132:0/1578265339 learned_addr learned my addr 172.21.15.132:0/1578265339 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:04.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.393+0000 7fa498e00640 1 --2- 172.21.15.132:0/1578265339 >> v2:172.21.15.134:3300/0 conn(0x7fa4940d0970 0x7fa49415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.394+0000 7fa493e00640 1 -- 172.21.15.132:0/1578265339 >> v2:172.21.15.134:3300/0 conn(0x7fa4940d0970 msgr2=0x7fa49415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:04.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.394+0000 7fa493e00640 1 --2- 172.21.15.132:0/1578265339 >> v2:172.21.15.134:3300/0 conn(0x7fa4940d0970 0x7fa49415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.394+0000 7fa493e00640 1 -- 172.21.15.132:0/1578265339 >> v1:172.21.15.134:6789/0 conn(0x7fa4940cb440 legacy=0x7fa4940c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.394+0000 7fa493e00640 1 -- 172.21.15.132:0/1578265339 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49415e8a0 con 0x7fa4940d0d50 2024-03-27T06:19:04.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.395+0000 7fa493e00640 1 --2- 172.21.15.132:0/1578265339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0d50 0x7fa494154aa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa49408d3e0 tx=0x7fa48002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6725e129b32d048b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:04.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.395+0000 7fa493400640 1 -- 172.21.15.132:0/1578265339 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480034070 con 0x7fa4940d0d50 2024-03-27T06:19:04.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.395+0000 7fa493400640 1 -- 172.21.15.132:0/1578265339 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa480030870 con 0x7fa4940d0d50 2024-03-27T06:19:04.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.395+0000 7fa493400640 1 -- 172.21.15.132:0/1578265339 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480030c40 con 0x7fa4940d0d50 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 -- 172.21.15.132:0/1578265339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0d50 msgr2=0x7fa494154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 --2- 172.21.15.132:0/1578265339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0d50 0x7fa494154aa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa49408d3e0 tx=0x7fa48002b9b0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 -- 172.21.15.132:0/1578265339 shutdown_connections 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 --2- 172.21.15.132:0/1578265339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0d50 0x7fa494154aa0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 --2- 172.21.15.132:0/1578265339 >> v2:172.21.15.134:3300/0 conn(0x7fa4940d0970 0x7fa49415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 -- 172.21.15.132:0/1578265339 >> 172.21.15.132:0/1578265339 conn(0x7fa49406dbd0 msgr2=0x7fa49406dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 -- 172.21.15.132:0/1578265339 shutdown_connections 2024-03-27T06:19:04.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.396+0000 7fa49b800640 1 -- 172.21.15.132:0/1578265339 wait complete. 2024-03-27T06:19:04.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.398+0000 7fa49b800640 1 Processor -- start 2024-03-27T06:19:04.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.398+0000 7fa49b800640 1 -- start start 2024-03-27T06:19:04.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4940cb440 0x7fa4940b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 0x7fa4940b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940cc970 con 0x7fa4940d0970 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940cc7f0 con 0x7fa4940cb440 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa49b800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa4940c9150 con 0x7fa4940d0d50 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51888/0 (socket says 172.21.15.132:51888) 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 -- 172.21.15.132:0/3926821023 learned_addr learned my addr 172.21.15.132:0/3926821023 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa498e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4940cb440 0x7fa4940b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa499800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 0x7fa4940b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 -- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 msgr2=0x7fa4940b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 0x7fa4940b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 -- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4940cb440 msgr2=0x7fa4940b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4940cb440 0x7fa4940b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa493e00640 1 -- 172.21.15.132:0/3926821023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4940b5390 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.399+0000 7fa499800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 0x7fa4940b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa493e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa480000c00 tx=0x7fa4800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480034040 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa480039d90 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480043610 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4940b5670 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.400+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4940b3450 con 0x7fa4940d0970 2024-03-27T06:19:04.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.402+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa494068840 con 0x7fa4940d0970 2024-03-27T06:19:04.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.402+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa480037070 con 0x7fa4940d0970 2024-03-27T06:19:04.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.403+0000 7fa490c00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 0x7fa47406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:04.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.404+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7fa4800ad910 con 0x7fa4940d0970 2024-03-27T06:19:04.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.404+0000 7fa498e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 0x7fa47406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:04.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.405+0000 7fa498e00640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 0x7fa47406ea20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa488009b40 tx=0x7fa488009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:04.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.409+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa48003e070 con 0x7fa4940d0970 2024-03-27T06:19:04.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.700+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4940cbff0 con 0x7fa4940d0970 2024-03-27T06:19:04.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.701+0000 7fa490c00640 1 -- 172.21.15.132:0/3926821023 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7fa48007b700 con 0x7fa4940d0970 2024-03-27T06:19:04.695 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:04.695 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:19:04.696 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:19:04.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 msgr2=0x7fa47406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 0x7fa47406ea20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa488009b40 tx=0x7fa488009190 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 msgr2=0x7fa4941f3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa480000c00 tx=0x7fa4800046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 shutdown_connections 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa4940d0d50 0x7fa4940b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4940d0970 0x7fa4941f3fa0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa47406c560 0x7fa47406ea20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 --2- 172.21.15.132:0/3926821023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4940cb440 0x7fa4940b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 >> 172.21.15.132:0/3926821023 conn(0x7fa49406dbd0 msgr2=0x7fa49406dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 shutdown_connections 2024-03-27T06:19:04.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:04.704+0000 7fa49b800640 1 -- 172.21.15.132:0/3926821023 wait complete. 2024-03-27T06:19:04.714 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-27T06:19:06.715 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:06.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.943+0000 7fdae9000640 1 Processor -- start 2024-03-27T06:19:06.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.943+0000 7fdae9000640 1 -- start start 2024-03-27T06:19:06.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae9000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdae41549a0 0x7fdae40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae9000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 0x7fdae415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae9000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae415de10 con 0x7fdae40cb870 2024-03-27T06:19:06.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae9000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fdae40cada0 con 0x7fdae415adb0 2024-03-27T06:19:06.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae9000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fdae415e0b0 con 0x7fdae41549a0 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae3e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 0x7fdae415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae3400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdae41549a0 0x7fdae40cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae2a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdae40cb870 0x7fdae406b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:51312/0 (socket says 172.21.15.132:51312) 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.945+0000 7fdae2a00640 1 -- 172.21.15.132:0/804297061 learned_addr learned my addr 172.21.15.132:0/804297061 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae3400640 1 -- 172.21.15.132:0/804297061 >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 msgr2=0x7fdae415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae3400640 1 --2- 172.21.15.132:0/804297061 >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 0x7fdae415b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae3400640 1 -- 172.21.15.132:0/804297061 >> v1:172.21.15.134:6789/0 conn(0x7fdae40cb870 legacy=0x7fdae406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae3400640 1 -- 172.21.15.132:0/804297061 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae415e390 con 0x7fdae41549a0 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae2000640 1 -- 172.21.15.132:0/804297061 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4194568963 0 0) 0x7fdae415de10 con 0x7fdae40cb870 2024-03-27T06:19:06.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.946+0000 7fdae3e00640 1 --2- 172.21.15.132:0/804297061 >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 0x7fdae415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:06.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.947+0000 7fdae3400640 1 --2- 172.21.15.132:0/804297061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae41549a0 0x7fdae40cd6a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdae40c73e0 tx=0x7fdad402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b5251a085911792 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:06.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.947+0000 7fdae2000640 1 -- 172.21.15.132:0/804297061 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4034070 con 0x7fdae41549a0 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.947+0000 7fdae2000640 1 -- 172.21.15.132:0/804297061 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad4030610 con 0x7fdae41549a0 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.947+0000 7fdae2000640 1 -- 172.21.15.132:0/804297061 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad40309e0 con 0x7fdae41549a0 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 -- 172.21.15.132:0/804297061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae41549a0 msgr2=0x7fdae40cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 --2- 172.21.15.132:0/804297061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae41549a0 0x7fdae40cd6a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdae40c73e0 tx=0x7fdad402b960 comp rx=0 tx=0).stop 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 -- 172.21.15.132:0/804297061 shutdown_connections 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 --2- 172.21.15.132:0/804297061 >> v2:172.21.15.132:3300/0 conn(0x7fdae415adb0 0x7fdae415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 --2- 172.21.15.132:0/804297061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae41549a0 0x7fdae40cd6a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 -- 172.21.15.132:0/804297061 >> 172.21.15.132:0/804297061 conn(0x7fdae406d8d0 msgr2=0x7fdae406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 -- 172.21.15.132:0/804297061 shutdown_connections 2024-03-27T06:19:06.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.948+0000 7fdae9000640 1 -- 172.21.15.132:0/804297061 wait complete. 2024-03-27T06:19:06.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.950+0000 7fdae9000640 1 Processor -- start 2024-03-27T06:19:06.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.951+0000 7fdae9000640 1 -- start start 2024-03-27T06:19:06.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 0x7fdae40c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdae415adb0 0x7fdae4156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdae415ec30 con 0x7fdae41549a0 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdae415e950 con 0x7fdae40cb870 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae9000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdae415b700 con 0x7fdae415adb0 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 0x7fdae40c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.952+0000 7fdae3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:55724/0 (socket says 172.21.15.132:55724) 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 -- 172.21.15.132:0/3544652244 learned_addr learned my addr 172.21.15.132:0/3544652244 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3e00640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdae415adb0 0x7fdae4156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 -- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdae415adb0 msgr2=0x7fdae4156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdae415adb0 0x7fdae4156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 -- 172.21.15.132:0/3544652244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 msgr2=0x7fdae40c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:06.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 0x7fdae40c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 -- 172.21.15.132:0/3544652244 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae40b36e0 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae2a00640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 0x7fdae40c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdae3400640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009890 tx=0x7fdad40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.953+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4034040 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.954+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad4030410 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.954+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4006e10 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.954+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae40b39c0 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.954+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae40b3f80 con 0x7fdae40cb870 2024-03-27T06:19:06.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.955+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae40686c0 con 0x7fdae40cb870 2024-03-27T06:19:06.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.957+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdad4037070 con 0x7fdae40cb870 2024-03-27T06:19:06.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.959+0000 7fdadb400640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 0x7fdac406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:06.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.959+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6024+0+0 (secure 0 0 0) 0x7fdad40ad7b0 con 0x7fdae40cb870 2024-03-27T06:19:06.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.959+0000 7fdae2a00640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 0x7fdac406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:06.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.959+0000 7fdae2a00640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 0x7fdac406e9f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdae40c63e0 tx=0x7fdad0007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:06.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:06.963+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fdad40305e0 con 0x7fdae40cb870 2024-03-27T06:19:07.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.223+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdae4158470 con 0x7fdae40cb870 2024-03-27T06:19:07.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.224+0000 7fdadb400640 1 -- 172.21.15.132:0/3544652244 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6888 (secure 0 0 0) 0x7fdad407bdb0 con 0x7fdae40cb870 2024-03-27T06:19:07.215 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:07.215 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:18:29.492359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4273,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.134:6839/3802035585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":3802035585},{"type":"v1","addr":"172.21.15.134:6839","nonce":3802035585}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4817,4718,4273]},"id":1}]} 2024-03-27T06:19:07.215 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 30 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 msgr2=0x7fdac406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 0x7fdac406e9f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdae40c63e0 tx=0x7fdad0007d50 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 msgr2=0x7fdae4159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009890 tx=0x7fdad40046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 shutdown_connections 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdae415adb0 0x7fdae4156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdae41549a0 0x7fdae40c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdac406c530 0x7fdac406e9f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 --2- 172.21.15.132:0/3544652244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdae40cb870 0x7fdae4159950 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 >> 172.21.15.132:0/3544652244 conn(0x7fdae406d8d0 msgr2=0x7fdae40cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 shutdown_connections 2024-03-27T06:19:07.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:07.227+0000 7fdae9000640 1 -- 172.21.15.132:0/3544652244 wait complete. 2024-03-27T06:19:07.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-03-27T06:19:09.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:09.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.485+0000 7f9d7ec00640 1 Processor -- start 2024-03-27T06:19:09.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.485+0000 7f9d7ec00640 1 -- start start 2024-03-27T06:19:09.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9d780cb870 0x7f9d7806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9d7806be10 0x7f9d7815adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f9d780cada0 con 0x7f9d780cb870 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f9d7815de10 con 0x7f9d7806be10 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7815e0b0 con 0x7f9d781549a0 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7ce00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9d780cb870 0x7f9d7806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.487+0000 7f9d7e200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9d7806be10 0x7f9d7815adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40366/0 (socket says 172.21.15.132:40366) 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7d800640 1 -- 172.21.15.132:0/1129957831 learned_addr learned my addr 172.21.15.132:0/1129957831 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7e200640 1 -- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 msgr2=0x7f9d780cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7e200640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7e200640 1 -- 172.21.15.132:0/1129957831 >> v2:172.21.15.132:3300/0 conn(0x7f9d780cb870 msgr2=0x7f9d7806b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7e200640 1 --2- 172.21.15.132:0/1129957831 >> v2:172.21.15.132:3300/0 conn(0x7f9d780cb870 0x7f9d7806b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7e200640 1 -- 172.21.15.132:0/1129957831 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7815e390 con 0x7f9d7806be10 2024-03-27T06:19:09.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.488+0000 7f9d7d800640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.490+0000 7f9d7e200640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d7806be10 0x7f9d7815adb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9d780cdf20 tx=0x7f9d7002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8ada42a1d36b2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.490+0000 7f9d77e00640 1 -- 172.21.15.132:0/1129957831 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70039070 con 0x7f9d7806be10 2024-03-27T06:19:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.490+0000 7f9d77e00640 1 -- 172.21.15.132:0/1129957831 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7002cc70 con 0x7f9d7806be10 2024-03-27T06:19:09.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.490+0000 7f9d77e00640 1 -- 172.21.15.132:0/1129957831 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70034660 con 0x7f9d7806be10 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 -- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d7806be10 msgr2=0x7f9d7815adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d7806be10 0x7f9d7815adb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9d780cdf20 tx=0x7f9d7002bec0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 -- 172.21.15.132:0/1129957831 shutdown_connections 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d7806be10 0x7f9d7815adb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/1129957831 >> v2:172.21.15.132:3300/0 conn(0x7f9d780cb870 0x7f9d7806b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/1129957831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d781549a0 0x7f9d780cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 -- 172.21.15.132:0/1129957831 >> 172.21.15.132:0/1129957831 conn(0x7f9d7806d8d0 msgr2=0x7f9d7806dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 -- 172.21.15.132:0/1129957831 shutdown_connections 2024-03-27T06:19:09.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.491+0000 7f9d7ec00640 1 -- 172.21.15.132:0/1129957831 wait complete. 2024-03-27T06:19:09.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.493+0000 7f9d7ec00640 1 Processor -- start 2024-03-27T06:19:09.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.493+0000 7f9d7ec00640 1 -- start start 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 0x7f9d7814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 0x7f9d78150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7815ec30 con 0x7f9d780cb870 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7815e950 con 0x7f9d781549a0 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ec00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7815a820 con 0x7f9d7806be10 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ce00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 0x7f9d7814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40378/0 (socket says 172.21.15.132:40378) 2024-03-27T06:19:09.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7d800640 1 -- 172.21.15.132:0/3110297772 learned_addr learned my addr 172.21.15.132:0/3110297772 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:09.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 0x7f9d78150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 -- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 msgr2=0x7f9d78159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 -- 172.21.15.132:0/3110297772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 msgr2=0x7f9d7814ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 0x7f9d7814ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7e200640 1 -- 172.21.15.132:0/3110297772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d78149930 con 0x7f9d781549a0 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7d800640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.495+0000 7f9d7ce00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 0x7f9d7814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d7e200640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 0x7f9d78150450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d70033000 tx=0x7f9d70004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70039050 con 0x7f9d781549a0 2024-03-27T06:19:09.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d70045070 con 0x7f9d781549a0 2024-03-27T06:19:09.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7815c3c0 con 0x7f9d781549a0 2024-03-27T06:19:09.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70009450 con 0x7f9d781549a0 2024-03-27T06:19:09.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.496+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7815c970 con 0x7f9d781549a0 2024-03-27T06:19:09.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.498+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d780686c0 con 0x7f9d781549a0 2024-03-27T06:19:09.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.499+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9d7003e850 con 0x7f9d781549a0 2024-03-27T06:19:09.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.501+0000 7f9d75600640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 0x7f9d5806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.501+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f9d700ae610 con 0x7f9d781549a0 2024-03-27T06:19:09.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.501+0000 7f9d7d800640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 0x7f9d5806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.502+0000 7f9d7d800640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 0x7f9d5806e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0113d0 tx=0x7f9d6c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.505+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9d7002ce20 con 0x7f9d781549a0 2024-03-27T06:19:09.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.745+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d780cafd0 con 0x7f9d781549a0 2024-03-27T06:19:09.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.747+0000 7f9d75600640 1 -- 172.21.15.132:0/3110297772 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6056 (secure 0 0 0) 0x7f9d7007c520 con 0x7f9d781549a0 2024-03-27T06:19:09.737 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:09.737 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:09.174938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 31 2024-03-27T06:19:09.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 msgr2=0x7f9d5806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 0x7f9d5806e9f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0113d0 tx=0x7f9d6c009240 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 msgr2=0x7f9d78150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 0x7f9d78150450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d70033000 tx=0x7f9d70004a40 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 shutdown_connections 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9d5806c530 0x7f9d5806e9f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d781549a0 0x7f9d78150450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d780cb870 0x7f9d7814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 --2- 172.21.15.132:0/3110297772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9d7806be10 0x7f9d78159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 >> 172.21.15.132:0/3110297772 conn(0x7f9d7806d8d0 msgr2=0x7f9d780cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.750+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 shutdown_connections 2024-03-27T06:19:09.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.751+0000 7f9d7ec00640 1 -- 172.21.15.132:0/3110297772 wait complete. 2024-03-27T06:19:09.759 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-03-27T06:19:09.760 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-27T06:19:09.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:09.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.968+0000 7f9eae000640 1 Processor -- start 2024-03-27T06:19:09.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.968+0000 7f9eae000640 1 -- start start 2024-03-27T06:19:09.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9eae000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 0x7f9ea80c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9eae000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9ea806cc80 0x7f9ea80c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9eae000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea80cdad0 con 0x7f9ea80c77f0 2024-03-27T06:19:09.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9eae000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea80cc210 con 0x7f9ea806cc80 2024-03-27T06:19:09.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9eae000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea815b690 con 0x7f9ea806c8a0 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9ea6a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9ea806cc80 0x7f9ea80c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9ea7400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 0x7f9ea80c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9ea7400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 0x7f9ea80c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:40384/0 (socket says 172.21.15.132:40384) 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.971+0000 7f9ea7400640 1 -- 172.21.15.132:0/2275665715 learned_addr learned my addr 172.21.15.132:0/2275665715 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.972+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2275665715 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 msgr2=0x7f9ea80c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.972+0000 7f9ea6a00640 1 --2- 172.21.15.132:0/2275665715 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 0x7f9ea80c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.972+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2275665715 >> v1:172.21.15.134:6789/0 conn(0x7f9ea80c77f0 legacy=0x7f9ea8154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.972+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2275665715 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea815b970 con 0x7f9ea806cc80 2024-03-27T06:19:09.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.973+0000 7f9ea6a00640 1 --2- 172.21.15.132:0/2275665715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea80c8860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e940097a0 tx=0x7f9e9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2341cf16e0b19f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.973+0000 7f9ea6000640 1 -- 172.21.15.132:0/2275665715 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940315a0 con 0x7f9ea806cc80 2024-03-27T06:19:09.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.973+0000 7f9ea6000640 1 -- 172.21.15.132:0/2275665715 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e94037070 con 0x7f9ea806cc80 2024-03-27T06:19:09.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.973+0000 7f9ea6000640 1 -- 172.21.15.132:0/2275665715 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94031960 con 0x7f9ea806cc80 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 -- 172.21.15.132:0/2275665715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 msgr2=0x7f9ea80c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 --2- 172.21.15.132:0/2275665715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea80c8860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e940097a0 tx=0x7f9e9402b840 comp rx=0 tx=0).stop 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 -- 172.21.15.132:0/2275665715 shutdown_connections 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 --2- 172.21.15.132:0/2275665715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea80c8860 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 --2- 172.21.15.132:0/2275665715 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea806c8a0 0x7f9ea80c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 -- 172.21.15.132:0/2275665715 >> 172.21.15.132:0/2275665715 conn(0x7f9ea80ccbe0 msgr2=0x7f9ea80ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 -- 172.21.15.132:0/2275665715 shutdown_connections 2024-03-27T06:19:09.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.974+0000 7f9eae000640 1 -- 172.21.15.132:0/2275665715 wait complete. 2024-03-27T06:19:09.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.975+0000 7f9eae000640 1 Processor -- start 2024-03-27T06:19:09.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.976+0000 7f9eae000640 1 -- start start 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.976+0000 7f9eae000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea806c8a0 0x7f9ea8158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.976+0000 7f9eae000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.976+0000 7f9eae000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea80c77f0 0x7f9ea81f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9eae000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea806d440 con 0x7f9ea806cc80 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9eae000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea806d2c0 con 0x7f9ea806c8a0 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9eae000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea80cd700 con 0x7f9ea80c77f0 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51926/0 (socket says 172.21.15.132:51926) 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2699674131 learned_addr learned my addr 172.21.15.132:0/2699674131 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea7400640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea806c8a0 0x7f9ea8158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea7e00640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea80c77f0 0x7f9ea81f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea80c77f0 msgr2=0x7f9ea81f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea80c77f0 0x7f9ea81f42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea806c8a0 msgr2=0x7f9ea8158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea806c8a0 0x7f9ea8158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.977+0000 7f9ea6a00640 1 -- 172.21.15.132:0/2699674131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea81498f0 con 0x7f9ea806cc80 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9ea6a00640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e94034040 tx=0x7f9e940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94032630 con 0x7f9ea806cc80 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e940327f0 con 0x7f9ea806cc80 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea8149bd0 con 0x7f9ea806cc80 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94032be0 con 0x7f9ea806cc80 2024-03-27T06:19:09.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.978+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea81f3af0 con 0x7f9ea806cc80 2024-03-27T06:19:09.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.979+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea8068840 con 0x7f9ea806cc80 2024-03-27T06:19:09.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.981+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9e9403b3f0 con 0x7f9ea806cc80 2024-03-27T06:19:09.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.982+0000 7f9e9f400640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 0x7f9e8806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:09.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.982+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f9e9402f030 con 0x7f9ea806cc80 2024-03-27T06:19:09.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.983+0000 7f9ea7400640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 0x7f9e8806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:09.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.983+0000 7f9ea7400640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 0x7f9e8806e9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009bd0 tx=0x7f9e98009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:09.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:09.985+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9e94040070 con 0x7f9ea806cc80 2024-03-27T06:19:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.238+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ea80cb920 con 0x7f9ea806cc80 2024-03-27T06:19:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.239+0000 7f9e9f400640 1 -- 172.21.15.132:0/2699674131 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6056 (secure 0 0 0) 0x7f9e9407b8b0 con 0x7f9ea806cc80 2024-03-27T06:19:10.229 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:10.229 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:09.174938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 31 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.241+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 msgr2=0x7f9e8806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.241+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 0x7f9e8806e9f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009bd0 tx=0x7f9e98009240 comp rx=0 tx=0).stop 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.241+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 msgr2=0x7f9ea815ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.241+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e94034040 tx=0x7f9e940046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 shutdown_connections 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9ea80c77f0 0x7f9ea81f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9ea806cc80 0x7f9ea815ea10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9e8806c530 0x7f9e8806e9f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 --2- 172.21.15.132:0/2699674131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea806c8a0 0x7f9ea8158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:10.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 >> 172.21.15.132:0/2699674131 conn(0x7f9ea80ccbe0 msgr2=0x7f9ea8158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:10.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 shutdown_connections 2024-03-27T06:19:10.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:10.242+0000 7f9eae000640 1 -- 172.21.15.132:0/2699674131 wait complete. 2024-03-27T06:19:10.250 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 21 }, "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_4904": { "addr": "172.21.15.134:6838/2682370774", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2682370774, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2682370774, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4904, "incarnation": 31, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:19:09.174938+0000", "qdb_cluster": [ 4718, 4817 ], "qdb_leader": 4817, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4904, "mds_1": 4817, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:19:12.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:12.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.488+0000 7efecfe00640 1 Processor -- start 2024-03-27T06:19:12.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.489+0000 7efecfe00640 1 -- start start 2024-03-27T06:19:12.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecfe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7efed01549a0 0x7efed00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecfe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efed00cb870 0x7efed006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecfe00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed015de10 con 0x7efed006be10 2024-03-27T06:19:12.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecfe00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7efed00cada0 con 0x7efed01549a0 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecfe00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7efed015e0b0 con 0x7efed00cb870 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7efed01549a0 0x7efed00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7efed01549a0 0x7efed00cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46946/0 (socket says 172.21.15.132:46946) 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efecea00640 1 -- 172.21.15.132:0/3257978093 learned_addr learned my addr 172.21.15.132:0/3257978093 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.490+0000 7efec6000640 1 --2- 172.21.15.132:0/3257978093 >> v2:172.21.15.134:3300/0 conn(0x7efed00cb870 0x7efed006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.491+0000 7efecea00640 1 -- 172.21.15.132:0/3257978093 >> v2:172.21.15.134:3300/0 conn(0x7efed00cb870 msgr2=0x7efed006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.491+0000 7efecea00640 1 --2- 172.21.15.132:0/3257978093 >> v2:172.21.15.134:3300/0 conn(0x7efed00cb870 0x7efed006b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.491+0000 7efecea00640 1 -- 172.21.15.132:0/3257978093 >> v1:172.21.15.134:6789/0 conn(0x7efed006be10 legacy=0x7efed015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.491+0000 7efecea00640 1 -- 172.21.15.132:0/3257978093 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed015e390 con 0x7efed01549a0 2024-03-27T06:19:12.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.493+0000 7efecea00640 1 --2- 172.21.15.132:0/3257978093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed00cd6a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efed00b2df0 tx=0x7efec002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dbe8f867c9dadf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:12.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.493+0000 7efece000640 1 -- 172.21.15.132:0/3257978093 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec002cbb0 con 0x7efed01549a0 2024-03-27T06:19:12.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.493+0000 7efece000640 1 -- 172.21.15.132:0/3257978093 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec002cd50 con 0x7efed01549a0 2024-03-27T06:19:12.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.493+0000 7efece000640 1 -- 172.21.15.132:0/3257978093 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0034610 con 0x7efed01549a0 2024-03-27T06:19:12.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 -- 172.21.15.132:0/3257978093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 msgr2=0x7efed00cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 --2- 172.21.15.132:0/3257978093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed00cd6a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efed00b2df0 tx=0x7efec002c010 comp rx=0 tx=0).stop 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 -- 172.21.15.132:0/3257978093 shutdown_connections 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 --2- 172.21.15.132:0/3257978093 >> v2:172.21.15.134:3300/0 conn(0x7efed00cb870 0x7efed006b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 --2- 172.21.15.132:0/3257978093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed00cd6a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 -- 172.21.15.132:0/3257978093 >> 172.21.15.132:0/3257978093 conn(0x7efed006d8d0 msgr2=0x7efed006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 -- 172.21.15.132:0/3257978093 shutdown_connections 2024-03-27T06:19:12.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.494+0000 7efecfe00640 1 -- 172.21.15.132:0/3257978093 wait complete. 2024-03-27T06:19:12.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.496+0000 7efecfe00640 1 Processor -- start 2024-03-27T06:19:12.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.497+0000 7efecfe00640 1 -- start start 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efed006be10 0x7efed00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efed00cb870 0x7efed0159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efed015ec30 con 0x7efed01549a0 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efed015e950 con 0x7efed006be10 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecfe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7efed015a820 con 0x7efed00cb870 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.498+0000 7efecf400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46960/0 (socket says 172.21.15.132:46960) 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 -- 172.21.15.132:0/1381269307 learned_addr learned my addr 172.21.15.132:0/1381269307 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efec6000640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efed00cb870 0x7efed0159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecea00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efed006be10 0x7efed00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 -- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efed00cb870 msgr2=0x7efed0159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efed00cb870 0x7efed0159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 -- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efed006be10 msgr2=0x7efed00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efed006be10 0x7efed00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 -- 172.21.15.132:0/1381269307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed01f1090 con 0x7efed01549a0 2024-03-27T06:19:12.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.499+0000 7efecf400640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efec800c400 tx=0x7efec800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.500+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8017070 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.500+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec80052d0 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.500+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8003810 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.500+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed01585b0 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.500+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed0158b70 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.501+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed00ca930 con 0x7efed01549a0 2024-03-27T06:19:12.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.502+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7efec801a070 con 0x7efed01549a0 2024-03-27T06:19:12.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.503+0000 7efec7400640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 0x7efeac06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:12.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.503+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7efec808abc0 con 0x7efed01549a0 2024-03-27T06:19:12.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.504+0000 7efecea00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 0x7efeac06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:12.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.504+0000 7efecea00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 0x7efeac06ea20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efec0035000 tx=0x7efec0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:12.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.507+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7efed00ca930 con 0x7efed01549a0 2024-03-27T06:19:12.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.820+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efed00686c0 con 0x7efed01549a0 2024-03-27T06:19:12.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.822+0000 7efec7400640 1 -- 172.21.15.132:0/1381269307 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6056 (secure 0 0 0) 0x7efec8058970 con 0x7efed01549a0 2024-03-27T06:19:12.812 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:12.812 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:09.174938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:12.812 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 31 2024-03-27T06:19:12.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.824+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 msgr2=0x7efeac06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.824+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 0x7efeac06ea20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efec0035000 tx=0x7efec0030000 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.824+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 msgr2=0x7efed01598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.824+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efec800c400 tx=0x7efec800c8c0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 shutdown_connections 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efed01549a0 0x7efed01598b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efed00cb870 0x7efed0159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7efeac06c560 0x7efeac06ea20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 --2- 172.21.15.132:0/1381269307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efed006be10 0x7efed00b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 >> 172.21.15.132:0/1381269307 conn(0x7efed006d8d0 msgr2=0x7efed00c5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 shutdown_connections 2024-03-27T06:19:12.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:12.825+0000 7efecfe00640 1 -- 172.21.15.132:0/1381269307 wait complete. 2024-03-27T06:19:12.832 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:14.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.043+0000 7fd6a6200640 1 Processor -- start 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.043+0000 7fd6a6200640 1 -- start start 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.044+0000 7fd6a6200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd6a006cf50 0x7fd6a006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.044+0000 7fd6a6200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a006d920 con 0x7fd6a01590a0 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.044+0000 7fd6a6200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a006b630 con 0x7fd6a0152950 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.044+0000 7fd6a6200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd6a0152d30 con 0x7fd6a006cf50 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69ea00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd6a01590a0 0x7fd6a0071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:40802/0 (socket says 172.21.15.132:40802) 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69ea00640 1 -- 172.21.15.132:0/304650748 learned_addr learned my addr 172.21.15.132:0/304650748 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69f400640 1 --1- 172.21.15.132:0/304650748 >> v1:172.21.15.134:6789/0 conn(0x7fd6a0152950 0x7fd6a0073550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:55566/0 (socket says 172.21.15.132:55566) 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69fe00640 1 --2- 172.21.15.132:0/304650748 >> v2:172.21.15.132:3300/0 conn(0x7fd6a006cf50 0x7fd6a006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 550478279 0 0) 0x7fd6a006b630 con 0x7fd6a0152950 2024-03-27T06:19:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd684003520 con 0x7fd6a0152950 2024-03-27T06:19:15.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854019516 0 0) 0x7fd6a006d920 con 0x7fd6a01590a0 2024-03-27T06:19:15.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6a006b630 con 0x7fd6a01590a0 2024-03-27T06:19:15.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69fe00640 1 -- 172.21.15.132:0/304650748 >> v1:172.21.15.134:6789/0 conn(0x7fd6a0152950 legacy=0x7fd6a0073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69fe00640 1 -- 172.21.15.132:0/304650748 >> v1:172.21.15.132:6789/0 conn(0x7fd6a01590a0 legacy=0x7fd6a0071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.045+0000 7fd69fe00640 1 -- 172.21.15.132:0/304650748 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a006b8d0 con 0x7fd6a006cf50 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.046+0000 7fd69fe00640 1 --2- 172.21.15.132:0/304650748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a006cf50 0x7fd6a006d3e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd698009d90 tx=0x7fd69802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65039396078fd9a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.047+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd698039070 con 0x7fd6a006cf50 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.047+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd69802cd00 con 0x7fd6a006cf50 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.047+0000 7fd69e000640 1 -- 172.21.15.132:0/304650748 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd698034640 con 0x7fd6a006cf50 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.047+0000 7fd6a6200640 1 -- 172.21.15.132:0/304650748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a006cf50 msgr2=0x7fd6a006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.047+0000 7fd6a6200640 1 --2- 172.21.15.132:0/304650748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a006cf50 0x7fd6a006d3e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd698009d90 tx=0x7fd69802bf50 comp rx=0 tx=0).stop 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.048+0000 7fd6a6200640 1 -- 172.21.15.132:0/304650748 shutdown_connections 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.048+0000 7fd6a6200640 1 --2- 172.21.15.132:0/304650748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a006cf50 0x7fd6a006d3e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.048+0000 7fd6a6200640 1 -- 172.21.15.132:0/304650748 >> 172.21.15.132:0/304650748 conn(0x7fd6a00cc060 msgr2=0x7fd6a00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.048+0000 7fd6a6200640 1 -- 172.21.15.132:0/304650748 shutdown_connections 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.048+0000 7fd6a6200640 1 -- 172.21.15.132:0/304650748 wait complete. 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.050+0000 7fd6a6200640 1 Processor -- start 2024-03-27T06:19:15.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.050+0000 7fd6a6200640 1 -- start start 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 0x7fd6a01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a006c170 con 0x7fd6a0152950 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a006be90 con 0x7fd6a01590a0 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd6a6200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a006c450 con 0x7fd6a006cf50 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69f400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 0x7fd6a01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46976/0 (socket says 172.21.15.132:46976) 2024-03-27T06:19:15.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:42798/0 (socket says 172.21.15.132:42798) 2024-03-27T06:19:15.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 -- 172.21.15.132:0/1812245471 learned_addr learned my addr 172.21.15.132:0/1812245471 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 -- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 msgr2=0x7fd6a01f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 0x7fd6a01f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 -- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 msgr2=0x7fd6a00c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69ea00640 1 -- 172.21.15.132:0/1812245471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a01541d0 con 0x7fd6a0152950 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69fe00640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.051+0000 7fd69f400640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 0x7fd6a01f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd69ea00640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c3c0 tx=0x7fd68c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c017070 con 0x7fd6a0152950 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd68c005ce0 con 0x7fd6a0152950 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a01544b0 con 0x7fd6a0152950 2024-03-27T06:19:15.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c007310 con 0x7fd6a0152950 2024-03-27T06:19:15.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.052+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a0154a70 con 0x7fd6a0152950 2024-03-27T06:19:15.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.053+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a00727b0 con 0x7fd6a0152950 2024-03-27T06:19:15.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.054+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd68c01a070 con 0x7fd6a0152950 2024-03-27T06:19:15.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.055+0000 7fd697400640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 0x7fd68006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:15.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.056+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fd68c08ac70 con 0x7fd6a0152950 2024-03-27T06:19:15.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.056+0000 7fd69f400640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 0x7fd68006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:15.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.056+0000 7fd69f400640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 0x7fd68006e9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd690000f60 tx=0x7fd690007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:15.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.060+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd6a00727b0 con 0x7fd6a0152950 2024-03-27T06:19:15.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.336+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6a0068840 con 0x7fd6a0152950 2024-03-27T06:19:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.337+0000 7fd697400640 1 -- 172.21.15.132:0/1812245471 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6056 (secure 0 0 0) 0x7fd68c058990 con 0x7fd6a0152950 2024-03-27T06:19:15.327 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:15.327 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:09.174938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:15.327 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 31 2024-03-27T06:19:15.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 msgr2=0x7fd68006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 0x7fd68006e9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd690000f60 tx=0x7fd690007cd0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 msgr2=0x7fd6a01f18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c3c0 tx=0x7fd68c00c880 comp rx=0 tx=0).stop 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 shutdown_connections 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd68006c530 0x7fd68006e9f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6a01590a0 0x7fd6a00c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6a0152950 0x7fd6a01f18e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 --2- 172.21.15.132:0/1812245471 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd6a006cf50 0x7fd6a01f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 >> 172.21.15.132:0/1812245471 conn(0x7fd6a00cc060 msgr2=0x7fd6a00c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 shutdown_connections 2024-03-27T06:19:15.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:15.340+0000 7fd6a6200640 1 -- 172.21.15.132:0/1812245471 wait complete. 2024-03-27T06:19:15.358 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:17.358 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.579+0000 7f5b06e00640 1 Processor -- start 2024-03-27T06:19:17.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.579+0000 7f5b06e00640 1 -- start start 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.581+0000 7f5b06e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5b0006c8a0 0x7f5b000c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.581+0000 7f5b06e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006cc80 0x7f5b000c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.581+0000 7f5b06e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b000cdad0 con 0x7f5b000c77f0 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.581+0000 7f5b06e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f5b000cc210 con 0x7f5b0006c8a0 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.581+0000 7f5b06e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5b0015b690 con 0x7f5b0006cc80 2024-03-27T06:19:17.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5b0006c8a0 0x7f5b000c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5b0006c8a0 0x7f5b000c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46982/0 (socket says 172.21.15.132:46982) 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 -- 172.21.15.132:0/2280103359 learned_addr learned my addr 172.21.15.132:0/2280103359 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5b04e00640 1 --1- 172.21.15.132:0/2280103359 >> v1:172.21.15.132:6789/0 conn(0x7f5b000c77f0 0x7f5b00154890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:40812/0 (socket says 172.21.15.132:40812) 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5aff400640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006cc80 0x7f5b000c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 -- 172.21.15.132:0/2280103359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006cc80 msgr2=0x7f5b000c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006cc80 0x7f5b000c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 -- 172.21.15.132:0/2280103359 >> v1:172.21.15.132:6789/0 conn(0x7f5b000c77f0 legacy=0x7f5b00154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5affe00640 1 -- 172.21.15.132:0/2280103359 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0015b970 con 0x7f5b0006c8a0 2024-03-27T06:19:17.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.582+0000 7f5afea00640 1 -- 172.21.15.132:0/2280103359 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614265116 0 0) 0x7f5b000cdad0 con 0x7f5b000c77f0 2024-03-27T06:19:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.584+0000 7f5affe00640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b0006c8a0 0x7f5b000c8320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b10ff6f85a24f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.584+0000 7f5afea00640 1 -- 172.21.15.132:0/2280103359 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0035020 con 0x7f5b0006c8a0 2024-03-27T06:19:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.584+0000 7f5afea00640 1 -- 172.21.15.132:0/2280103359 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af0031640 con 0x7f5b0006c8a0 2024-03-27T06:19:17.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.584+0000 7f5afea00640 1 -- 172.21.15.132:0/2280103359 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af002d040 con 0x7f5b0006c8a0 2024-03-27T06:19:17.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 -- 172.21.15.132:0/2280103359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b0006c8a0 msgr2=0x7f5b000c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b0006c8a0 0x7f5b000c8320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af002b8f0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 -- 172.21.15.132:0/2280103359 shutdown_connections 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006cc80 0x7f5b000c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 --2- 172.21.15.132:0/2280103359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b0006c8a0 0x7f5b000c8320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 -- 172.21.15.132:0/2280103359 >> 172.21.15.132:0/2280103359 conn(0x7f5b000ccbe0 msgr2=0x7f5b000ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 -- 172.21.15.132:0/2280103359 shutdown_connections 2024-03-27T06:19:17.576 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.585+0000 7f5b06e00640 1 -- 172.21.15.132:0/2280103359 wait complete. 2024-03-27T06:19:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.587+0000 7f5b06e00640 1 Processor -- start 2024-03-27T06:19:17.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.587+0000 7f5b06e00640 1 -- start start 2024-03-27T06:19:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0006cc80 0x7f5b000c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b000c77f0 0x7f5b000c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0006d440 con 0x7f5b000c77f0 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0006d2c0 con 0x7f5b0006cc80 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b06e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5b000cd700 con 0x7f5b0006c8a0 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:53076/0 (socket says 172.21.15.132:53076) 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 -- 172.21.15.132:0/3514782210 learned_addr learned my addr 172.21.15.132:0/3514782210 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5b04e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b000c77f0 0x7f5b000c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5aff400640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0006cc80 0x7f5b000c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 -- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0006cc80 msgr2=0x7f5b000c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0006cc80 0x7f5b000c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.579 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 -- 172.21.15.132:0/3514782210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b000c77f0 msgr2=0x7f5b000c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b000c77f0 0x7f5b000c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.588+0000 7f5affe00640 1 -- 172.21.15.132:0/3514782210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b000c4130 con 0x7f5b0006c8a0 2024-03-27T06:19:17.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5affe00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5af002b930 tx=0x7f5af00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0040070 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af00399f0 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0035040 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b000c4410 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.589+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b000c49d0 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.590+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b00068840 con 0x7f5b0006c8a0 2024-03-27T06:19:17.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.591+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5af004b060 con 0x7f5b0006c8a0 2024-03-27T06:19:17.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.592+0000 7f5af7e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 0x7f5ae006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:17.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.592+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f5af00ad110 con 0x7f5b0006c8a0 2024-03-27T06:19:17.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.593+0000 7f5aff400640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 0x7f5ae006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:17.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.593+0000 7f5aff400640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 0x7f5ae006ea20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b000c90b0 tx=0x7f5aec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:17.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.597+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5af007aec0 con 0x7f5b0006c8a0 2024-03-27T06:19:17.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.900+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b000c81c0 con 0x7f5b0006c8a0 2024-03-27T06:19:17.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.901+0000 7f5af7e00640 1 -- 172.21.15.132:0/3514782210 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6058 (secure 0 0 0) 0x7f5af007f1c0 con 0x7f5b0006c8a0 2024-03-27T06:19:17.891 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:17.891 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:16.389014+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:resolve","state_seq":12,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4817,4718]},"id":1}]} 2024-03-27T06:19:17.891 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 32 2024-03-27T06:19:17.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 msgr2=0x7f5ae006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 0x7f5ae006ea20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b000c90b0 tx=0x7f5aec009190 comp rx=0 tx=0).stop 2024-03-27T06:19:17.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 msgr2=0x7f5b001551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5af002b930 tx=0x7f5af00046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 shutdown_connections 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b000c77f0 0x7f5b000c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5ae006c560 0x7f5ae006ea20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0006cc80 0x7f5b000c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 --2- 172.21.15.132:0/3514782210 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5b0006c8a0 0x7f5b001551c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.903+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 >> 172.21.15.132:0/3514782210 conn(0x7f5b000ccbe0 msgr2=0x7f5b00150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.904+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 shutdown_connections 2024-03-27T06:19:17.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:17.904+0000 7f5b06e00640 1 -- 172.21.15.132:0/3514782210 wait complete. 2024-03-27T06:19:17.911 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:19.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:20.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.151+0000 7ff8e1200640 1 Processor -- start 2024-03-27T06:19:20.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.151+0000 7ff8e1200640 1 -- start start 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8e1200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff8dc15adb0 0x7ff8dc15b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8e1200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc15de10 con 0x7ff8dc1549a0 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8e1200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc15e0b0 con 0x7ff8dc0cd6a0 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8e1200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc0cada0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8db400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ff8dc1549a0 0x7ff8dc0cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:40828/0 (socket says 172.21.15.132:40828) 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8db400640 1 -- 172.21.15.132:0/605807194 learned_addr learned my addr 172.21.15.132:0/605807194 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:20.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.154+0000 7ff8dbe00640 1 --2- 172.21.15.132:0/605807194 >> v2:172.21.15.132:3300/0 conn(0x7ff8dc15adb0 0x7ff8dc15b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:20.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050278785 0 0) 0x7ff8dc15de10 con 0x7ff8dc1549a0 2024-03-27T06:19:20.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8c0003520 con 0x7ff8dc1549a0 2024-03-27T06:19:20.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555254096 0 0) 0x7ff8dc15e0b0 con 0x7ff8dc0cd6a0 2024-03-27T06:19:20.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8dc15de10 con 0x7ff8dc0cd6a0 2024-03-27T06:19:20.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8dbe00640 1 -- 172.21.15.132:0/605807194 >> v1:172.21.15.134:6789/0 conn(0x7ff8dc0cd6a0 legacy=0x7ff8dc06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8dbe00640 1 -- 172.21.15.132:0/605807194 >> v1:172.21.15.132:6789/0 conn(0x7ff8dc1549a0 legacy=0x7ff8dc0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.155+0000 7ff8dbe00640 1 -- 172.21.15.132:0/605807194 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc15e390 con 0x7ff8dc15adb0 2024-03-27T06:19:20.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.157+0000 7ff8dbe00640 1 --2- 172.21.15.132:0/605807194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc15b1c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009d90 tx=0x7ff8d402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd043efbcfc7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:20.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.157+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4039070 con 0x7ff8dc15adb0 2024-03-27T06:19:20.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.157+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d402cd80 con 0x7ff8dc15adb0 2024-03-27T06:19:20.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.157+0000 7ff8da000640 1 -- 172.21.15.132:0/605807194 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4034640 con 0x7ff8dc15adb0 2024-03-27T06:19:20.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 -- 172.21.15.132:0/605807194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 msgr2=0x7ff8dc15b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 --2- 172.21.15.132:0/605807194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc15b1c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009d90 tx=0x7ff8d402bfd0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 -- 172.21.15.132:0/605807194 shutdown_connections 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 --2- 172.21.15.132:0/605807194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc15b1c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 -- 172.21.15.132:0/605807194 >> 172.21.15.132:0/605807194 conn(0x7ff8dc06d8d0 msgr2=0x7ff8dc06dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 -- 172.21.15.132:0/605807194 shutdown_connections 2024-03-27T06:19:20.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.158+0000 7ff8e1200640 1 -- 172.21.15.132:0/605807194 wait complete. 2024-03-27T06:19:20.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.159+0000 7ff8e1200640 1 Processor -- start 2024-03-27T06:19:20.149 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 -- start start 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8dc0cd6a0 0x7ff8dc159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 0x7ff8dc15d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc15e950 con 0x7ff8dc15adb0 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cdad0 con 0x7ff8dc0cd6a0 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.160+0000 7ff8e1200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc15ec30 con 0x7ff8dc1549a0 2024-03-27T06:19:20.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:47002/0 (socket says 172.21.15.132:47002) 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 -- 172.21.15.132:0/3123269689 learned_addr learned my addr 172.21.15.132:0/3123269689 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8db400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8dc0cd6a0 0x7ff8dc159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8daa00640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 0x7ff8dc15d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 -- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 msgr2=0x7ff8dc15d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 0x7ff8dc15d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 -- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8dc0cd6a0 msgr2=0x7ff8dc159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8dc0cd6a0 0x7ff8dc159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8dbe00640 1 -- 172.21.15.132:0/3123269689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc1f06d0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.161+0000 7ff8daa00640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 0x7ff8dc15d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8dbe00640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009eb0 tx=0x7ff8d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4039050 con 0x7ff8dc15adb0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d4045070 con 0x7ff8dc15adb0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4009450 con 0x7ff8dc15adb0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc158dd0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.162+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc1f0c30 con 0x7ff8dc15adb0 2024-03-27T06:19:20.153 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.163+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc0686c0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.164+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff8d403e850 con 0x7ff8dc15adb0 2024-03-27T06:19:20.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.165+0000 7ff8d3400640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 0x7ff8bc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:20.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.165+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7ff8d40add50 con 0x7ff8dc15adb0 2024-03-27T06:19:20.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.165+0000 7ff8db400640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 0x7ff8bc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:20.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.166+0000 7ff8db400640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 0x7ff8bc06ea20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000f60 tx=0x7ff8cc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:20.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.174+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff8d404e020 con 0x7ff8dc15adb0 2024-03-27T06:19:20.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.460+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8dc14a9f0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.462+0000 7ff8d3400640 1 -- 172.21.15.132:0/3123269689 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6058 (secure 0 0 0) 0x7ff8dc1f06d0 con 0x7ff8dc15adb0 2024-03-27T06:19:20.452 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:20.452 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:16.389014+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:resolve","state_seq":12,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4817,4718]},"id":1}]} 2024-03-27T06:19:20.452 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 32 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.464+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 msgr2=0x7ff8bc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.464+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 0x7ff8bc06ea20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000f60 tx=0x7ff8cc007d30 comp rx=0 tx=0).stop 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.464+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 msgr2=0x7ff8dc158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.464+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009eb0 tx=0x7ff8d4004a40 comp rx=0 tx=0).stop 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 shutdown_connections 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8dc15adb0 0x7ff8dc158340 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff8dc1549a0 0x7ff8dc15d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff8bc06c560 0x7ff8bc06ea20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 --2- 172.21.15.132:0/3123269689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8dc0cd6a0 0x7ff8dc159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:20.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 >> 172.21.15.132:0/3123269689 conn(0x7ff8dc06d8d0 msgr2=0x7ff8dc0cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:20.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 shutdown_connections 2024-03-27T06:19:20.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:20.465+0000 7ff8e1200640 1 -- 172.21.15.132:0/3123269689 wait complete. 2024-03-27T06:19:20.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:22.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:22.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.716+0000 7fa79fe00640 1 Processor -- start 2024-03-27T06:19:22.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.716+0000 7fa79fe00640 1 -- start start 2024-03-27T06:19:22.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.718+0000 7fa79fe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa7a01549a0 0x7fa7a00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.718+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79fe00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a015de10 con 0x7fa7a006be10 2024-03-27T06:19:22.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79fe00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa7a00cada0 con 0x7fa7a01549a0 2024-03-27T06:19:22.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79fe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a015e0b0 con 0x7fa7a00cb870 2024-03-27T06:19:22.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79ea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa7a01549a0 0x7fa7a00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79f400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa7a006be10 0x7fa7a015adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54928/0 (socket says 172.21.15.132:54928) 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79f400640 1 -- 172.21.15.132:0/2213868713 learned_addr learned my addr 172.21.15.132:0/2213868713 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79e000640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79ea00640 1 -- 172.21.15.132:0/2213868713 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 msgr2=0x7fa7a006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79ea00640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a006b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79ea00640 1 -- 172.21.15.132:0/2213868713 >> v1:172.21.15.132:6789/0 conn(0x7fa7a006be10 legacy=0x7fa7a015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.719+0000 7fa79ea00640 1 -- 172.21.15.132:0/2213868713 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a015e390 con 0x7fa7a01549a0 2024-03-27T06:19:22.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79ea00640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a00cd6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7900097a0 tx=0x7fa79002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04d779c1a9c2920 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:22.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79d600640 1 -- 172.21.15.132:0/2213868713 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790034070 con 0x7fa7a01549a0 2024-03-27T06:19:22.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79d600640 1 -- 172.21.15.132:0/2213868713 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa790030800 con 0x7fa7a01549a0 2024-03-27T06:19:22.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79d600640 1 -- 172.21.15.132:0/2213868713 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790030bd0 con 0x7fa7a01549a0 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 -- 172.21.15.132:0/2213868713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 msgr2=0x7fa7a00cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a00cd6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7900097a0 tx=0x7fa79002b940 comp rx=0 tx=0).stop 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 -- 172.21.15.132:0/2213868713 shutdown_connections 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a006b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 --2- 172.21.15.132:0/2213868713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a00cd6a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.721+0000 7fa79fe00640 1 -- 172.21.15.132:0/2213868713 >> 172.21.15.132:0/2213868713 conn(0x7fa7a006d8d0 msgr2=0x7fa7a006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.722+0000 7fa79fe00640 1 -- 172.21.15.132:0/2213868713 shutdown_connections 2024-03-27T06:19:22.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.722+0000 7fa79fe00640 1 -- 172.21.15.132:0/2213868713 wait complete. 2024-03-27T06:19:22.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.723+0000 7fa79fe00640 1 Processor -- start 2024-03-27T06:19:22.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.723+0000 7fa79fe00640 1 -- start start 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7a006be10 0x7fa7a0147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a01f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a015ec30 con 0x7fa7a01549a0 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a015e950 con 0x7fa7a006be10 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.724+0000 7fa79fe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a015a820 con 0x7fa7a00cb870 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:41204/0 (socket says 172.21.15.132:41204) 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 -- 172.21.15.132:0/1300073801 learned_addr learned my addr 172.21.15.132:0/1300073801 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79ea00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7a006be10 0x7fa7a0147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79e000640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a01f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 -- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 msgr2=0x7fa7a01f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a01f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 -- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7a006be10 msgr2=0x7fa7a0147710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7a006be10 0x7fa7a0147710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 -- 172.21.15.132:0/1300073801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a0147d50 con 0x7fa7a01549a0 2024-03-27T06:19:22.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.725+0000 7fa79f400640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa79800ca80 tx=0x7fa79800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.726+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798017070 con 0x7fa7a01549a0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.726+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79800ac30 con 0x7fa7a01549a0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.726+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a00c9e40 con 0x7fa7a01549a0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.726+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798012600 con 0x7fa7a01549a0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.726+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a00ca400 con 0x7fa7a01549a0 2024-03-27T06:19:22.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.727+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a006dcb0 con 0x7fa7a01549a0 2024-03-27T06:19:22.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.728+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa79801a070 con 0x7fa7a01549a0 2024-03-27T06:19:22.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.729+0000 7fa796a00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 0x7fa78006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:22.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.729+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fa79808b5f0 con 0x7fa7a01549a0 2024-03-27T06:19:22.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.729+0000 7fa79ea00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 0x7fa78006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:22.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.730+0000 7fa79ea00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 0x7fa78006e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa790036000 tx=0x7fa79003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:22.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.737+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa7a006dcb0 con 0x7fa7a01549a0 2024-03-27T06:19:22.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.974+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7a00cbe30 con 0x7fa7a01549a0 2024-03-27T06:19:22.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.975+0000 7fa796a00640 1 -- 172.21.15.132:0/1300073801 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6058 (secure 0 0 0) 0x7fa79800ae20 con 0x7fa7a01549a0 2024-03-27T06:19:22.965 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:22.965 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:16.389014+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:resolve","state_seq":12,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4817,4718]},"id":1}]} 2024-03-27T06:19:22.965 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 32 2024-03-27T06:19:22.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 msgr2=0x7fa78006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 0x7fa78006e9f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa790036000 tx=0x7fa79003b040 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 msgr2=0x7fa7a01f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa79800ca80 tx=0x7fa79800cf40 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 shutdown_connections 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7a01549a0 0x7fa7a01f2e40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa7a00cb870 0x7fa7a01f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa78006c530 0x7fa78006e9f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 --2- 172.21.15.132:0/1300073801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7a006be10 0x7fa7a0147710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.977+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 >> 172.21.15.132:0/1300073801 conn(0x7fa7a006d8d0 msgr2=0x7fa7a0159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.978+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 shutdown_connections 2024-03-27T06:19:22.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:22.978+0000 7fa79fe00640 1 -- 172.21.15.132:0/1300073801 wait complete. 2024-03-27T06:19:22.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4718": { "addr": "172.21.15.132:6837/1424864911", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6835", "nonce": 1424864911, "type": "v2" }, { "addr": "172.21.15.132:6837", "nonce": 1424864911, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4718, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 21 }, "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_4904": { "addr": "172.21.15.134:6838/2682370774", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2682370774, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2682370774, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4904, "incarnation": 31, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:resolve", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:19:16.389014+0000", "qdb_cluster": [ 4817, 4718 ], "qdb_leader": 4817, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4904, "mds_1": 4817, "mds_2": 4718 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:19:24.988 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:25.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.200+0000 7f2dfca00640 1 Processor -- start 2024-03-27T06:19:25.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.200+0000 7f2dfca00640 1 -- start start 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2dfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df81549a0 0x7f2df80cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2dfca00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2df815adb0 0x7f2df815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2dfca00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2df815de10 con 0x7f2df80cb870 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2dfca00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2df80cada0 con 0x7f2df815adb0 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2dfca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2df815b720 con 0x7f2df81549a0 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2df5e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f2df80cb870 0x7f2df806b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54944/0 (socket says 172.21.15.132:54944) 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2df5e00640 1 -- 172.21.15.132:0/4124498824 learned_addr learned my addr 172.21.15.132:0/4124498824 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:25.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2df7200640 1 --2- 172.21.15.132:0/4124498824 >> v2:172.21.15.132:3300/0 conn(0x7f2df815adb0 0x7f2df815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.202+0000 7f2df6800640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df81549a0 0x7f2df80cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df5400640 1 -- 172.21.15.132:0/4124498824 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850425241 0 0) 0x7f2df815de10 con 0x7f2df80cb870 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df5400640 1 -- 172.21.15.132:0/4124498824 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ddc003520 con 0x7f2df80cb870 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df7200640 1 -- 172.21.15.132:0/4124498824 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df81549a0 msgr2=0x7f2df80cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df7200640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df81549a0 0x7f2df80cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df7200640 1 -- 172.21.15.132:0/4124498824 >> v1:172.21.15.132:6789/0 conn(0x7f2df80cb870 legacy=0x7f2df806b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.203+0000 7f2df7200640 1 -- 172.21.15.132:0/4124498824 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df815e0b0 con 0x7f2df815adb0 2024-03-27T06:19:25.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.204+0000 7f2df7200640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df815adb0 0x7f2df815b1e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009d90 tx=0x7f2df002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90542f134c6ef5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.204+0000 7f2df5400640 1 -- 172.21.15.132:0/4124498824 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0039070 con 0x7f2df815adb0 2024-03-27T06:19:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.204+0000 7f2df5400640 1 -- 172.21.15.132:0/4124498824 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df002cce0 con 0x7f2df815adb0 2024-03-27T06:19:25.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.204+0000 7f2df5400640 1 -- 172.21.15.132:0/4124498824 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0034660 con 0x7f2df815adb0 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 -- 172.21.15.132:0/4124498824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df815adb0 msgr2=0x7f2df815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df815adb0 0x7f2df815b1e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009d90 tx=0x7f2df002bf30 comp rx=0 tx=0).stop 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 -- 172.21.15.132:0/4124498824 shutdown_connections 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df815adb0 0x7f2df815b1e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 --2- 172.21.15.132:0/4124498824 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df81549a0 0x7f2df80cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 -- 172.21.15.132:0/4124498824 >> 172.21.15.132:0/4124498824 conn(0x7f2df806d8d0 msgr2=0x7f2df806dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 -- 172.21.15.132:0/4124498824 shutdown_connections 2024-03-27T06:19:25.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.205+0000 7f2dfca00640 1 -- 172.21.15.132:0/4124498824 wait complete. 2024-03-27T06:19:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.207+0000 7f2dfca00640 1 Processor -- start 2024-03-27T06:19:25.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.207+0000 7f2dfca00640 1 -- start start 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2df81549a0 0x7f2df8152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 0x7f2df81531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2df815e950 con 0x7f2df80cb870 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2df815e670 con 0x7f2df81549a0 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2dfca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2df815ec30 con 0x7f2df815adb0 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2df6800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2df6800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:41224/0 (socket says 172.21.15.132:41224) 2024-03-27T06:19:25.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2df6800640 1 -- 172.21.15.132:0/3247679020 learned_addr learned my addr 172.21.15.132:0/3247679020 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:25.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2df7200640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 0x7f2df81531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.208+0000 7f2df5e00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2df81549a0 0x7f2df8152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 -- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 msgr2=0x7f2df81531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 0x7f2df81531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 -- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2df81549a0 msgr2=0x7f2df8152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2df81549a0 0x7f2df8152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 -- 172.21.15.132:0/3247679020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df815c5b0 con 0x7f2df80cb870 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df7200640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 0x7f2df81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2df6800640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2de800e9e0 tx=0x7f2de800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8017070 con 0x7f2df80cb870 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de800c9d0 con 0x7f2df80cb870 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df815c890 con 0x7f2df80cb870 2024-03-27T06:19:25.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de800cdc0 con 0x7f2df80cb870 2024-03-27T06:19:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.209+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df815ce50 con 0x7f2df80cb870 2024-03-27T06:19:25.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.211+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df80686c0 con 0x7f2df80cb870 2024-03-27T06:19:25.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.213+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2de801a070 con 0x7f2df80cb870 2024-03-27T06:19:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.214+0000 7f2deea00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 0x7f2dd806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.214+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f2de808bb40 con 0x7f2df80cb870 2024-03-27T06:19:25.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.215+0000 7f2df5e00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 0x7f2dd806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:25.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.215+0000 7f2df5e00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 0x7f2dd806e9f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2df815b3c0 tx=0x7f2de4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:25.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.219+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2de8059980 con 0x7f2df80cb870 2024-03-27T06:19:25.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.591+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2df806c160 con 0x7f2df80cb870 2024-03-27T06:19:25.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.592+0000 7f2deea00640 1 -- 172.21.15.132:0/3247679020 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6060 (secure 0 0 0) 0x7f2de805dc80 con 0x7f2df80cb870 2024-03-27T06:19:25.583 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:25.583 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:22.305743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:25.583 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 33 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 msgr2=0x7f2dd806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 0x7f2dd806e9f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2df815b3c0 tx=0x7f2de4009040 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 msgr2=0x7f2df8153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2de800e9e0 tx=0x7f2de800eea0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 shutdown_connections 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2df815adb0 0x7f2df81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2dd806c530 0x7f2dd806e9f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2df81549a0 0x7f2df8152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 --2- 172.21.15.132:0/3247679020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df80cb870 0x7f2df8153d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 >> 172.21.15.132:0/3247679020 conn(0x7f2df806d8d0 msgr2=0x7f2df806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:25.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 shutdown_connections 2024-03-27T06:19:25.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:25.596+0000 7f2dfca00640 1 -- 172.21.15.132:0/3247679020 wait complete. 2024-03-27T06:19:25.605 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:27.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:27.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.822+0000 7f547c800640 1 Processor -- start 2024-03-27T06:19:27.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.822+0000 7f547c800640 1 -- start start 2024-03-27T06:19:27.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f547c800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 0x7f547415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:27.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f547c800640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547406ba60 con 0x7f547415bf30 2024-03-27T06:19:27.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f547c800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547406bd40 con 0x7f547406c3d0 2024-03-27T06:19:27.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f547c800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f547415ba80 con 0x7f5474154e00 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f5479400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f547415bf30 0x7f54740c8d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54954/0 (socket says 172.21.15.132:54954) 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f5479400640 1 -- 172.21.15.132:0/3986387509 learned_addr learned my addr 172.21.15.132:0/3986387509 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.824+0000 7f5479e00640 1 --2- 172.21.15.132:0/3986387509 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 0x7f547415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.825+0000 7f5479e00640 1 -- 172.21.15.132:0/3986387509 >> v1:172.21.15.134:6789/0 conn(0x7f547406c3d0 legacy=0x7f54740c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.825+0000 7f5479e00640 1 -- 172.21.15.132:0/3986387509 >> v1:172.21.15.132:6789/0 conn(0x7f547415bf30 legacy=0x7f54740c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.825+0000 7f5479e00640 1 -- 172.21.15.132:0/3986387509 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54740c8230 con 0x7f5474154e00 2024-03-27T06:19:27.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.825+0000 7f5478a00640 1 -- 172.21.15.132:0/3986387509 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 295419919 0 0) 0x7f547406bd40 con 0x7f547406c3d0 2024-03-27T06:19:27.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.826+0000 7f5479e00640 1 --2- 172.21.15.132:0/3986387509 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 0x7f547415b540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f54680097d0 tx=0x7f546802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e6b3db6cb952ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:27.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.826+0000 7f5478a00640 1 -- 172.21.15.132:0/3986387509 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468037070 con 0x7f5474154e00 2024-03-27T06:19:27.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.826+0000 7f5478a00640 1 -- 172.21.15.132:0/3986387509 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5468030710 con 0x7f5474154e00 2024-03-27T06:19:27.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.826+0000 7f5478a00640 1 -- 172.21.15.132:0/3986387509 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468030ae0 con 0x7f5474154e00 2024-03-27T06:19:27.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 -- 172.21.15.132:0/3986387509 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 msgr2=0x7f547415b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:27.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 --2- 172.21.15.132:0/3986387509 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 0x7f547415b540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f54680097d0 tx=0x7f546802b770 comp rx=0 tx=0).stop 2024-03-27T06:19:27.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 -- 172.21.15.132:0/3986387509 shutdown_connections 2024-03-27T06:19:27.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 --2- 172.21.15.132:0/3986387509 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5474154e00 0x7f547415b540 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:27.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 -- 172.21.15.132:0/3986387509 >> 172.21.15.132:0/3986387509 conn(0x7f54740d0150 msgr2=0x7f54740d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:27.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 -- 172.21.15.132:0/3986387509 shutdown_connections 2024-03-27T06:19:27.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.827+0000 7f547c800640 1 -- 172.21.15.132:0/3986387509 wait complete. 2024-03-27T06:19:27.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.829+0000 7f547c800640 1 Processor -- start 2024-03-27T06:19:27.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.829+0000 7f547c800640 1 -- start start 2024-03-27T06:19:27.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:27.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 0x7f547414da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:27.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f547415bf30 0x7f54741f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:27.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54741551e0 con 0x7f5474154e00 2024-03-27T06:19:27.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54740c87f0 con 0x7f547406c3d0 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547c800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f54740c7c60 con 0x7f547415bf30 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f5479400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 0x7f547414da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f5479e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f547a800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f547415bf30 0x7f54741f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f5479e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:57556/0 (socket says 172.21.15.132:57556) 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f5479e00640 1 -- 172.21.15.132:0/1713951990 learned_addr learned my addr 172.21.15.132:0/1713951990 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.830+0000 7f5479e00640 1 -- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f547415bf30 msgr2=0x7f54741f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479e00640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f547415bf30 0x7f54741f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479e00640 1 -- 172.21.15.132:0/1713951990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 msgr2=0x7f547414da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479e00640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 0x7f547414da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479e00640 1 -- 172.21.15.132:0/1713951990 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741f3df0 con 0x7f547406c3d0 2024-03-27T06:19:27.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479400640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 0x7f547414da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:19:27.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f5479e00640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f54680097d0 tx=0x7f54680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:27.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468042070 con 0x7f547406c3d0 2024-03-27T06:19:27.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546803e930 con 0x7f547406c3d0 2024-03-27T06:19:27.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468037040 con 0x7f547406c3d0 2024-03-27T06:19:27.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5474158a60 con 0x7f547406c3d0 2024-03-27T06:19:27.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.831+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5474159000 con 0x7f547406c3d0 2024-03-27T06:19:27.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.832+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5474068840 con 0x7f547406c3d0 2024-03-27T06:19:27.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.833+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5468045070 con 0x7f547406c3d0 2024-03-27T06:19:27.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.834+0000 7f546e000640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 0x7f545806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:27.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.834+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f54680ad540 con 0x7f547406c3d0 2024-03-27T06:19:27.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.835+0000 7f5479400640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 0x7f545806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:27.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.835+0000 7f5479400640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 0x7f545806e9f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f54740ceff0 tx=0x7f5464007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:27.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:27.839+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f546807b380 con 0x7f547406c3d0 2024-03-27T06:19:28.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.101+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54740ca3c0 con 0x7f547406c3d0 2024-03-27T06:19:28.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.102+0000 7f546e000640 1 -- 172.21.15.132:0/1713951990 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6060 (secure 0 0 0) 0x7f546807f680 con 0x7f547406c3d0 2024-03-27T06:19:28.092 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:28.092 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:22.305743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4718,4817]},"id":1}]} 2024-03-27T06:19:28.092 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 33 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.104+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 msgr2=0x7f545806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.104+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 0x7f545806e9f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f54740ceff0 tx=0x7f5464007e20 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.104+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 msgr2=0x7f5474159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.104+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f54680097d0 tx=0x7f54680046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 shutdown_connections 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f547415bf30 0x7f54741f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5474154e00 0x7f547414da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f545806c530 0x7f545806e9f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 --2- 172.21.15.132:0/1713951990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f547406c3d0 0x7f5474159da0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 >> 172.21.15.132:0/1713951990 conn(0x7f54740d0150 msgr2=0x7f547415c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:28.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 shutdown_connections 2024-03-27T06:19:28.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:28.105+0000 7f547c800640 1 -- 172.21.15.132:0/1713951990 wait complete. 2024-03-27T06:19:28.112 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:30.113 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:30.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.328+0000 7f8e01e00640 1 Processor -- start 2024-03-27T06:19:30.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.328+0000 7f8e01e00640 1 -- start start 2024-03-27T06:19:30.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.329+0000 7f8e01e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8dfc06ee50 0x7f8dfc06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.329+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc06e5a0 0x7f8dfc06fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.329+0000 7f8e01e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dfc15aac0 con 0x7f8dfc154940 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.329+0000 7f8e01e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f8dfc15b640 con 0x7f8dfc06ee50 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.329+0000 7f8e01e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8dfc06f9a0 con 0x7f8dfc06e5a0 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8dfc06ee50 0x7f8dfc06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8dfc06ee50 0x7f8dfc06e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:41232/0 (socket says 172.21.15.132:41232) 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2495672647 learned_addr learned my addr 172.21.15.132:0/2495672647 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:30.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8e01400640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc06e5a0 0x7f8dfc06fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2495672647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc06e5a0 msgr2=0x7f8dfc06fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc06e5a0 0x7f8dfc06fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2495672647 >> v1:172.21.15.134:6789/0 conn(0x7f8dfc154940 legacy=0x7f8dfc0c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2495672647 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc15ada0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.330+0000 7f8dfb400640 1 -- 172.21.15.132:0/2495672647 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495865367 0 0) 0x7f8dfc15aac0 con 0x7f8dfc154940 2024-03-27T06:19:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.331+0000 7f8dfbe00640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc06e060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688bcd66c286869c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.332+0000 7f8dfb400640 1 -- 172.21.15.132:0/2495672647 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de802f840 con 0x7f8dfc06ee50 2024-03-27T06:19:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.332+0000 7f8dfb400640 1 -- 172.21.15.132:0/2495672647 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de802f9e0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.332+0000 7f8dfb400640 1 -- 172.21.15.132:0/2495672647 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de802fdb0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.332+0000 7f8e01e00640 1 -- 172.21.15.132:0/2495672647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 msgr2=0x7f8dfc06e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.332+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc06e060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de802ba20 comp rx=0 tx=0).stop 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 -- 172.21.15.132:0/2495672647 shutdown_connections 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc06e5a0 0x7f8dfc06fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2495672647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc06e060 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 -- 172.21.15.132:0/2495672647 >> 172.21.15.132:0/2495672647 conn(0x7f8dfc0ccf20 msgr2=0x7f8dfc0cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 -- 172.21.15.132:0/2495672647 shutdown_connections 2024-03-27T06:19:30.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.333+0000 7f8e01e00640 1 -- 172.21.15.132:0/2495672647 wait complete. 2024-03-27T06:19:30.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.335+0000 7f8e01e00640 1 Processor -- start 2024-03-27T06:19:30.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.336+0000 7f8e01e00640 1 -- start start 2024-03-27T06:19:30.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dfc06e5a0 0x7f8dfc1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc154940 0x7f8dfc1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc0c8030 con 0x7f8dfc06ee50 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc15b360 con 0x7f8dfc06e5a0 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8dfc15a710 con 0x7f8dfc154940 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8dfbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8dfbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:41242/0 (socket says 172.21.15.132:41242) 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2091033182 learned_addr learned my addr 172.21.15.132:0/2091033182 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e00a00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dfc06e5a0 0x7f8dfc1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.337+0000 7f8e01400640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc154940 0x7f8dfc1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc154940 msgr2=0x7f8dfc1517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc154940 0x7f8dfc1517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dfc06e5a0 msgr2=0x7f8dfc1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dfc06e5a0 0x7f8dfc1f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 -- 172.21.15.132:0/2091033182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc1509a0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8dfbe00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8037bc0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8046070 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.338+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8041400 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.339+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dfc0c7050 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.339+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dfc150f20 con 0x7f8dfc06ee50 2024-03-27T06:19:30.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.340+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dfc06f620 con 0x7f8dfc06ee50 2024-03-27T06:19:30.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.341+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8de8040070 con 0x7f8dfc06ee50 2024-03-27T06:19:30.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.342+0000 7f8df8c00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 0x7f8ddc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:30.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.343+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f8de80aded0 con 0x7f8dfc06ee50 2024-03-27T06:19:30.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.343+0000 7f8e00a00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 0x7f8ddc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:30.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.343+0000 7f8e00a00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 0x7f8ddc06e9f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8df0000f60 tx=0x7f8df0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:30.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.347+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f8dfc06f620 con 0x7f8dfc06ee50 2024-03-27T06:19:30.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.648+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dfc158360 con 0x7f8dfc06ee50 2024-03-27T06:19:30.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.649+0000 7f8df8c00640 1 -- 172.21.15.132:0/2091033182 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6057 (secure 0 0 0) 0x7f8de8033030 con 0x7f8dfc06ee50 2024-03-27T06:19:30.639 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:30.639 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:28.294316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4817,4718]},"id":1}]} 2024-03-27T06:19:30.640 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 34 2024-03-27T06:19:30.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 msgr2=0x7f8ddc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 0x7f8ddc06e9f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8df0000f60 tx=0x7f8df0009040 comp rx=0 tx=0).stop 2024-03-27T06:19:30.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 msgr2=0x7f8dfc0c2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:30.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8004730 comp rx=0 tx=0).stop 2024-03-27T06:19:30.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 shutdown_connections 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dfc154940 0x7f8dfc1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dfc06ee50 0x7f8dfc0c2ba0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8ddc06c530 0x7f8ddc06e9f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 --2- 172.21.15.132:0/2091033182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dfc06e5a0 0x7f8dfc1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 >> 172.21.15.132:0/2091033182 conn(0x7f8dfc0ccf20 msgr2=0x7f8dfc0c1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 shutdown_connections 2024-03-27T06:19:30.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:30.653+0000 7f8e01e00640 1 -- 172.21.15.132:0/2091033182 wait complete. 2024-03-27T06:19:30.669 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:32.670 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:32.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.879+0000 7f41e0e00640 1 Processor -- start 2024-03-27T06:19:32.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.879+0000 7f41e0e00640 1 -- start start 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41e0e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41dc0ca160 0x7f41dc15b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41e0e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41dc154fe0 con 0x7f41dc06cbb0 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41e0e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41dc0ca540 con 0x7f41dc0cb5b0 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41e0e00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f41dc0caa80 con 0x7f41dc0ca160 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41d9e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41dc0ca160 0x7f41dc15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41da800640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f41dc06cbb0 0x7f41dc06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:51456/0 (socket says 172.21.15.132:51456) 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41d9400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f41dc0cb5b0 0x7f41dc070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:41590/0 (socket says 172.21.15.132:41590) 2024-03-27T06:19:32.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.882+0000 7f41da800640 1 -- 172.21.15.132:0/2581831517 learned_addr learned my addr 172.21.15.132:0/2581831517 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:32.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.883+0000 7f41d9e00640 1 -- 172.21.15.132:0/2581831517 >> v1:172.21.15.134:6789/0 conn(0x7f41dc0cb5b0 legacy=0x7f41dc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:32.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.883+0000 7f41d9e00640 1 -- 172.21.15.132:0/2581831517 >> v1:172.21.15.132:6789/0 conn(0x7f41dc06cbb0 legacy=0x7f41dc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:32.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.883+0000 7f41d9e00640 1 -- 172.21.15.132:0/2581831517 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc0cdfd0 con 0x7f41dc0ca160 2024-03-27T06:19:32.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.884+0000 7f41d9e00640 1 --2- 172.21.15.132:0/2581831517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc0ca160 0x7f41dc15b320 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097a0 tx=0x7f41cc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed7f458e47658acb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:32.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.884+0000 7f41d8a00640 1 -- 172.21.15.132:0/2581831517 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc034070 con 0x7f41dc0ca160 2024-03-27T06:19:32.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.884+0000 7f41d8a00640 1 -- 172.21.15.132:0/2581831517 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41cc030610 con 0x7f41dc0ca160 2024-03-27T06:19:32.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.884+0000 7f41d8a00640 1 -- 172.21.15.132:0/2581831517 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0309e0 con 0x7f41dc0ca160 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 -- 172.21.15.132:0/2581831517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc0ca160 msgr2=0x7f41dc15b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 --2- 172.21.15.132:0/2581831517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc0ca160 0x7f41dc15b320 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097a0 tx=0x7f41cc02b960 comp rx=0 tx=0).stop 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 -- 172.21.15.132:0/2581831517 shutdown_connections 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 --2- 172.21.15.132:0/2581831517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc0ca160 0x7f41dc15b320 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 -- 172.21.15.132:0/2581831517 >> 172.21.15.132:0/2581831517 conn(0x7f41dc073120 msgr2=0x7f41dc073500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 -- 172.21.15.132:0/2581831517 shutdown_connections 2024-03-27T06:19:32.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.885+0000 7f41e0e00640 1 -- 172.21.15.132:0/2581831517 wait complete. 2024-03-27T06:19:32.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.886+0000 7f41e0e00640 1 Processor -- start 2024-03-27T06:19:32.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.887+0000 7f41e0e00640 1 -- start start 2024-03-27T06:19:32.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 0x7f41dc1f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:32.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 0x7f41dc0c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:32.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41dc0cb5b0 0x7f41dc0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:32.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0ce850 con 0x7f41dc0ca160 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0ce570 con 0x7f41dc06cbb0 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41e0e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f41dc0ceb30 con 0x7f41dc0cb5b0 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 0x7f41dc0c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 0x7f41dc0c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:54122/0 (socket says 172.21.15.132:54122) 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 0x7f41dc1f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9400640 1 -- 172.21.15.132:0/1979972572 learned_addr learned my addr 172.21.15.132:0/1979972572 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41da800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41dc0cb5b0 0x7f41dc0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 -- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41dc0cb5b0 msgr2=0x7f41dc0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41dc0cb5b0 0x7f41dc0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 -- 172.21.15.132:0/1979972572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 msgr2=0x7f41dc0c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 0x7f41dc0c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.888+0000 7f41d9e00640 1 -- 172.21.15.132:0/1979972572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc14faa0 con 0x7f41dc06cbb0 2024-03-27T06:19:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41d9e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 0x7f41dc1f19a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41cc033000 tx=0x7f41cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc034040 con 0x7f41dc06cbb0 2024-03-27T06:19:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41cc03acc0 con 0x7f41dc06cbb0 2024-03-27T06:19:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc044630 con 0x7f41dc06cbb0 2024-03-27T06:19:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc14fd60 con 0x7f41dc06cbb0 2024-03-27T06:19:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.889+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc14eb70 con 0x7f41dc06cbb0 2024-03-27T06:19:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.891+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc068840 con 0x7f41dc06cbb0 2024-03-27T06:19:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.891+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f41cc04b060 con 0x7f41dc06cbb0 2024-03-27T06:19:32.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.892+0000 7f41d2000640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 0x7f41bc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:32.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.892+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f41cc0ad670 con 0x7f41dc06cbb0 2024-03-27T06:19:32.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.892+0000 7f41d9400640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 0x7f41bc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:32.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.893+0000 7f41d9400640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 0x7f41bc06e9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0cce50 tx=0x7f41c8007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:32.887 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:32.897+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f41cc03f070 con 0x7f41dc06cbb0 2024-03-27T06:19:33.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.132+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41dc0c61a0 con 0x7f41dc06cbb0 2024-03-27T06:19:33.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.133+0000 7f41d2000640 1 -- 172.21.15.132:0/1979972572 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6057 (secure 0 0 0) 0x7f41cc07b4b0 con 0x7f41dc06cbb0 2024-03-27T06:19:33.123 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:33.123 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:28.294316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4817,"qdb_cluster":[4817,4718]},"id":1}]} 2024-03-27T06:19:33.124 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 34 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.135+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 msgr2=0x7f41bc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 0x7f41bc06e9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0cce50 tx=0x7f41c8007f00 comp rx=0 tx=0).stop 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 msgr2=0x7f41dc1f19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 0x7f41dc1f19a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41cc033000 tx=0x7f41cc0046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 shutdown_connections 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f41dc0cb5b0 0x7f41dc0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41dc0ca160 0x7f41dc0c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f41bc06c530 0x7f41bc06e9f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:33.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 --2- 172.21.15.132:0/1979972572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41dc06cbb0 0x7f41dc1f19a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:33.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 >> 172.21.15.132:0/1979972572 conn(0x7f41dc073120 msgr2=0x7f41dc157a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:33.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 shutdown_connections 2024-03-27T06:19:33.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:33.136+0000 7f41e0e00640 1 -- 172.21.15.132:0/1979972572 wait complete. 2024-03-27T06:19:33.147 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:19:34.310 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:19:34.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:19:35.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:35.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.367+0000 7fb1a9a00640 1 Processor -- start 2024-03-27T06:19:35.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.368+0000 7fb1a9a00640 1 -- start start 2024-03-27T06:19:35.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.370+0000 7fb1a9a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb1a4152950 0x7fb1a41590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.370+0000 7fb1a9a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb1a4073550 0x7fb1a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.370+0000 7fb1a9a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a406b630 con 0x7fb1a4073970 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.370+0000 7fb1a9a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a4152d30 con 0x7fb1a4073550 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.370+0000 7fb1a9a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a406b8d0 con 0x7fb1a4152950 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a3400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fb1a4073970 0x7fb1a406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:51470/0 (socket says 172.21.15.132:51470) 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a3400640 1 -- 172.21.15.132:0/1737814301 learned_addr learned my addr 172.21.15.132:0/1737814301 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2000640 1 --2- 172.21.15.132:0/1737814301 >> v2:172.21.15.132:3300/0 conn(0x7fb1a4073550 0x7fb1a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/1737814301 >> v2:172.21.15.134:3300/0 conn(0x7fb1a4152950 0x7fb1a41590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2a00640 1 -- 172.21.15.132:0/1737814301 >> v2:172.21.15.132:3300/0 conn(0x7fb1a4073550 msgr2=0x7fb1a4071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:35.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/1737814301 >> v2:172.21.15.132:3300/0 conn(0x7fb1a4073550 0x7fb1a4071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2a00640 1 -- 172.21.15.132:0/1737814301 >> v1:172.21.15.132:6789/0 conn(0x7fb1a4073970 legacy=0x7fb1a406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.371+0000 7fb1a2a00640 1 -- 172.21.15.132:0/1737814301 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a406bbb0 con 0x7fb1a4152950 2024-03-27T06:19:35.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.373+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/1737814301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a41590a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b009edf377a524a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:35.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.373+0000 7fb1a1600640 1 -- 172.21.15.132:0/1737814301 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194035020 con 0x7fb1a4152950 2024-03-27T06:19:35.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.373+0000 7fb1a1600640 1 -- 172.21.15.132:0/1737814301 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194031830 con 0x7fb1a4152950 2024-03-27T06:19:35.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.373+0000 7fb1a1600640 1 -- 172.21.15.132:0/1737814301 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194031c00 con 0x7fb1a4152950 2024-03-27T06:19:35.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 -- 172.21.15.132:0/1737814301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 msgr2=0x7fb1a41590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.370 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/1737814301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a41590a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b8d0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 -- 172.21.15.132:0/1737814301 shutdown_connections 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/1737814301 >> v2:172.21.15.132:3300/0 conn(0x7fb1a4073550 0x7fb1a4071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/1737814301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a41590a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 -- 172.21.15.132:0/1737814301 >> 172.21.15.132:0/1737814301 conn(0x7fb1a40cc060 msgr2=0x7fb1a40cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 -- 172.21.15.132:0/1737814301 shutdown_connections 2024-03-27T06:19:35.371 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.374+0000 7fb1a9a00640 1 -- 172.21.15.132:0/1737814301 wait complete. 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.378+0000 7fb1a9a00640 1 Processor -- start 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.378+0000 7fb1a9a00640 1 -- start start 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb1a4073970 0x7fb1a41e4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a414ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a406c450 con 0x7fb1a4073550 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a406c170 con 0x7fb1a4152950 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a9a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a406d660 con 0x7fb1a4073970 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a2a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:54140/0 (socket says 172.21.15.132:54140) 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.380+0000 7fb1a2a00640 1 -- 172.21.15.132:0/3340250807 learned_addr learned my addr 172.21.15.132:0/3340250807 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2000640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb1a4073970 0x7fb1a41e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a3400640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a414ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 -- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb1a4073970 msgr2=0x7fb1a41e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb1a4073970 0x7fb1a41e4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 -- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 msgr2=0x7fb1a414ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a414ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 -- 172.21.15.132:0/3340250807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a41584c0 con 0x7fb1a4073550 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.381+0000 7fb1a2a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.382+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194040070 con 0x7fb1a4073550 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.382+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb19403e070 con 0x7fb1a4073550 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.382+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a40caa40 con 0x7fb1a4073550 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.382+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194035040 con 0x7fb1a4073550 2024-03-27T06:19:35.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.382+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41e37a0 con 0x7fb1a4073550 2024-03-27T06:19:35.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.383+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a4068840 con 0x7fb1a4073550 2024-03-27T06:19:35.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.385+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb194038020 con 0x7fb1a4073550 2024-03-27T06:19:35.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.386+0000 7fb19aa00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 0x7fb18406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:35.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.386+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fb1940acc60 con 0x7fb1a4073550 2024-03-27T06:19:35.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.387+0000 7fb1a2000640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 0x7fb18406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:35.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.387+0000 7fb1a2000640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 0x7fb18406ea20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1a415c5f0 tx=0x7fb190009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:35.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.391+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb194031800 con 0x7fb1a4073550 2024-03-27T06:19:35.632 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.642+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1a41e3970 con 0x7fb1a4073550 2024-03-27T06:19:35.633 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.643+0000 7fb19aa00640 1 -- 172.21.15.132:0/3340250807 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6062 (secure 0 0 0) 0x7fb19407aa10 con 0x7fb1a4073550 2024-03-27T06:19:35.633 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:35.633 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4718,4817,4904]},"id":1}]} 2024-03-27T06:19:35.633 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 35 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 msgr2=0x7fb18406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 0x7fb18406ea20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1a415c5f0 tx=0x7fb190009190 comp rx=0 tx=0).stop 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 msgr2=0x7fb1a41e3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 shutdown_connections 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb18406c560 0x7fb18406ea20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4152950 0x7fb1a414ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb1a4073970 0x7fb1a41e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 --2- 172.21.15.132:0/3340250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1a4073550 0x7fb1a41e3cc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:35.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 >> 172.21.15.132:0/3340250807 conn(0x7fb1a40cc060 msgr2=0x7fb1a40c8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:35.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 shutdown_connections 2024-03-27T06:19:35.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:35.646+0000 7fb1a9a00640 1 -- 172.21.15.132:0/3340250807 wait complete. 2024-03-27T06:19:35.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4273 2024-03-27T06:19:35.652 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.f 2024-03-27T06:19:38.944 INFO:tasks.workunit.client.0.smithi132.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-03-27T06:19:38.944 INFO:tasks.workunit.client.0.smithi132.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-03-27T06:19:38.947 INFO:tasks.workunit.client.0.smithi132.stderr:fatal: early EOF 2024-03-27T06:19:38.949 INFO:tasks.workunit.client.0.smithi132.stderr:fatal: fetch-pack: invalid index-pack output 2024-03-27T06:19:39.116 INFO:tasks.workunit.client.0.smithi132.stderr:+ retry 2024-03-27T06:19:39.116 INFO:tasks.workunit.client.0.smithi132.stderr:+ rm -rf ceph 2024-03-27T06:19:39.119 INFO:tasks.workunit.client.0.smithi132.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-03-27T06:19:39.126 INFO:tasks.workunit.client.0.smithi132.stderr:Cloning into 'ceph'... 2024-03-27T06:19:39.653 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-03-27T06:19:39.653 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-27T06:19:39.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-27T06:19:39.656 INFO:tasks.ceph.mds.f:Started 2024-03-27T06:19:39.656 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:39.732 INFO:tasks.ceph.mds.f.smithi134.stdout:starting mds.f at 2024-03-27T06:19:39.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.856+0000 7f1bfa400640 1 Processor -- start 2024-03-27T06:19:39.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.856+0000 7f1bfa400640 1 -- start start 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.857+0000 7f1bfa400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 0x7f1bf40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.857+0000 7f1bfa400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.857+0000 7f1bfa400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f1bf406be10 0x7f1bf415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bfa400640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf40cada0 con 0x7f1bf406be10 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bfa400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf415de10 con 0x7f1bf41549a0 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bfa400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf415e0b0 con 0x7f1bf40cb870 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf9000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 0x7f1bf40cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf9000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 0x7f1bf40cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36326/0 (socket says 172.21.15.132:36326) 2024-03-27T06:19:39.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:53598/0 (socket says 172.21.15.132:53598) 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4061196904 learned_addr learned my addr 172.21.15.132:0/4061196904 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf9a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f1bf406be10 0x7f1bf415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4061196904 >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 msgr2=0x7f1bf40cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4061196904 >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 0x7f1bf40cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4061196904 >> v2:172.21.15.132:3300/0 conn(0x7f1bf406be10 msgr2=0x7f1bf415adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4061196904 >> v2:172.21.15.132:3300/0 conn(0x7f1bf406be10 0x7f1bf415adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.858+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4061196904 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf415e390 con 0x7f1bf40cb870 2024-03-27T06:19:39.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.859+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4061196904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40c73e0 tx=0x7f1be002b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b21936206542c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:39.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bf3400640 1 -- 172.21.15.132:0/4061196904 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0034070 con 0x7f1bf40cb870 2024-03-27T06:19:39.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bf3400640 1 -- 172.21.15.132:0/4061196904 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be0030860 con 0x7f1bf40cb870 2024-03-27T06:19:39.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bf3400640 1 -- 172.21.15.132:0/4061196904 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0030c30 con 0x7f1bf40cb870 2024-03-27T06:19:39.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 -- 172.21.15.132:0/4061196904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 msgr2=0x7f1bf406b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4061196904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40c73e0 tx=0x7f1be002b9a0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 -- 172.21.15.132:0/4061196904 shutdown_connections 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4061196904 >> v2:172.21.15.132:3300/0 conn(0x7f1bf406be10 0x7f1bf415adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4061196904 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf40cb870 0x7f1bf406b8d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4061196904 >> v2:172.21.15.134:3300/0 conn(0x7f1bf41549a0 0x7f1bf40cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 -- 172.21.15.132:0/4061196904 >> 172.21.15.132:0/4061196904 conn(0x7f1bf406d8d0 msgr2=0x7f1bf406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 -- 172.21.15.132:0/4061196904 shutdown_connections 2024-03-27T06:19:39.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.860+0000 7f1bfa400640 1 -- 172.21.15.132:0/4061196904 wait complete. 2024-03-27T06:19:39.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.862+0000 7f1bfa400640 1 Processor -- start 2024-03-27T06:19:39.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.862+0000 7f1bfa400640 1 -- start start 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 0x7f1bf4152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 0x7f1bf40ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf41549a0 0x7f1bf40ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf415ec30 con 0x7f1bf41549a0 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf415e950 con 0x7f1bf40cb870 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bfa400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf415a820 con 0x7f1bf406be10 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bf9000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 0x7f1bf4152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 0x7f1bf40ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bf9000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 0x7f1bf4152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:53610/0 (socket says 172.21.15.132:53610) 2024-03-27T06:19:39.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.863+0000 7f1bf9000640 1 -- 172.21.15.132:0/4032062603 learned_addr learned my addr 172.21.15.132:0/4032062603 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf9a00640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf41549a0 0x7f1bf40ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 msgr2=0x7f1bf4152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 0x7f1bf4152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4032062603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf41549a0 msgr2=0x7f1bf40ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf41549a0 0x7f1bf40ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 -- 172.21.15.132:0/4032062603 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf4150c20 con 0x7f1bf40cb870 2024-03-27T06:19:39.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf3e00640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 0x7f1bf40ce3b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1be0033000 tx=0x7f1be0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.864+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0048070 con 0x7f1bf40cb870 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.865+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be0030e40 con 0x7f1bf40cb870 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.865+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0034040 con 0x7f1bf40cb870 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.865+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf40d02c0 con 0x7f1bf40cb870 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.865+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf40d0880 con 0x7f1bf40cb870 2024-03-27T06:19:39.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.866+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf40686c0 con 0x7f1bf40cb870 2024-03-27T06:19:39.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.867+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1be0037070 con 0x7f1bf40cb870 2024-03-27T06:19:39.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.868+0000 7f1bf0c00640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 0x7f1bd406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:39.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.869+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f1be00ad3b0 con 0x7f1bf40cb870 2024-03-27T06:19:39.859 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.869+0000 7f1bf9000640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 0x7f1bd406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:39.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.870+0000 7f1bf9000640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 0x7f1bd406e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c00 tx=0x7f1be8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:39.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:39.873+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f1be007b310 con 0x7f1bf40cb870 2024-03-27T06:19:40.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.111+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bf40b5550 con 0x7f1bf40cb870 2024-03-27T06:19:40.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.112+0000 7f1bf0c00640 1 -- 172.21.15.132:0/4032062603 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6062 (secure 0 0 0) 0x7f1be007f610 con 0x7f1bf40cb870 2024-03-27T06:19:40.103 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:40.103 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4718,4817,4904]},"id":1}]} 2024-03-27T06:19:40.103 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 35 2024-03-27T06:19:40.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 msgr2=0x7f1bd406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:40.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 0x7f1bd406e9f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c00 tx=0x7f1be8009190 comp rx=0 tx=0).stop 2024-03-27T06:19:40.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 msgr2=0x7f1bf40ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 0x7f1bf40ce3b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1be0033000 tx=0x7f1be0004730 comp rx=0 tx=0).stop 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 shutdown_connections 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf41549a0 0x7f1bf40ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1bd406c530 0x7f1bd406e9f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf40cb870 0x7f1bf40ce3b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 --2- 172.21.15.132:0/4032062603 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1bf406be10 0x7f1bf4152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 >> 172.21.15.132:0/4032062603 conn(0x7f1bf406d8d0 msgr2=0x7f1bf40cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 shutdown_connections 2024-03-27T06:19:40.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:40.114+0000 7f1bfa400640 1 -- 172.21.15.132:0/4032062603 wait complete. 2024-03-27T06:19:40.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-03-27T06:19:41.530 INFO:tasks.ceph.mds.b.smithi134.stderr:2024-03-27T06:19:41.540+0000 7fa4708d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-27T06:19:41.533 INFO:tasks.ceph.mds.b.smithi134.stderr:2024-03-27T06:19:41.540+0000 7fa4708d3640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-27T06:19:41.533 INFO:tasks.ceph.mds.b.smithi134.stderr:2024-03-27T06:19:41.540+0000 7fa4708d3640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-27T06:19:42.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:19:42.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.338+0000 7fa270c00640 1 Processor -- start 2024-03-27T06:19:42.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.338+0000 7fa270c00640 1 -- start start 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.340+0000 7fa270c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa26c0c8a30 0x7fa26c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.340+0000 7fa270c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c0cdad0 con 0x7fa26c06cc80 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.340+0000 7fa270c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c15b690 con 0x7fa26c06c8a0 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.340+0000 7fa270c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa26c0cc210 con 0x7fa26c0c8a30 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa269e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa26c06c8a0 0x7fa26c0c8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53316/0 (socket says 172.21.15.132:53316) 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa269e00640 1 -- 172.21.15.132:0/1300683405 learned_addr learned my addr 172.21.15.132:0/1300683405 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:19:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa26a800640 1 --2- 172.21.15.132:0/1300683405 >> v2:172.21.15.134:3300/0 conn(0x7fa26c0c8a30 0x7fa26c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672320653 0 0) 0x7fa26c15b690 con 0x7fa26c06c8a0 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa250003520 con 0x7fa26c06c8a0 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803933761 0 0) 0x7fa26c0cdad0 con 0x7fa26c06cc80 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa26a800640 1 -- 172.21.15.132:0/1300683405 >> v1:172.21.15.134:6789/0 conn(0x7fa26c06c8a0 legacy=0x7fa26c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa26a800640 1 -- 172.21.15.132:0/1300683405 >> v1:172.21.15.132:6789/0 conn(0x7fa26c06cc80 legacy=0x7fa26c154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.341+0000 7fa26a800640 1 -- 172.21.15.132:0/1300683405 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c15b970 con 0x7fa26c0c8a30 2024-03-27T06:19:42.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.343+0000 7fa26a800640 1 --2- 172.21.15.132:0/1300683405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c0c8a30 0x7fa26c154fa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa264009d90 tx=0x7fa26402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=720f7a2ee4011bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:42.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.343+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264039070 con 0x7fa26c0c8a30 2024-03-27T06:19:42.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.343+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26402cc60 con 0x7fa26c0c8a30 2024-03-27T06:19:42.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.343+0000 7fa268a00640 1 -- 172.21.15.132:0/1300683405 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264034660 con 0x7fa26c0c8a30 2024-03-27T06:19:42.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 -- 172.21.15.132:0/1300683405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c0c8a30 msgr2=0x7fa26c154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 --2- 172.21.15.132:0/1300683405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c0c8a30 0x7fa26c154fa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa264009d90 tx=0x7fa26402beb0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 -- 172.21.15.132:0/1300683405 shutdown_connections 2024-03-27T06:19:42.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 --2- 172.21.15.132:0/1300683405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c0c8a30 0x7fa26c154fa0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 -- 172.21.15.132:0/1300683405 >> 172.21.15.132:0/1300683405 conn(0x7fa26c0ccbe0 msgr2=0x7fa26c0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:42.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 -- 172.21.15.132:0/1300683405 shutdown_connections 2024-03-27T06:19:42.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.344+0000 7fa270c00640 1 -- 172.21.15.132:0/1300683405 wait complete. 2024-03-27T06:19:42.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.347+0000 7fa270c00640 1 Processor -- start 2024-03-27T06:19:42.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.347+0000 7fa270c00640 1 -- start start 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa26c06cc80 0x7fa26c0c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 0x7fa26c1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c06d440 con 0x7fa26c0c8a30 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c06d2c0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa270c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c0cd700 con 0x7fa26c06cc80 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:59230/0 (socket says 172.21.15.132:59230) 2024-03-27T06:19:42.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 -- 172.21.15.132:0/2536982389 learned_addr learned my addr 172.21.15.132:0/2536982389 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269400640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa26c06cc80 0x7fa26c0c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa26a800640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 0x7fa26c1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 -- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa26c06cc80 msgr2=0x7fa26c0c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa26c06cc80 0x7fa26c0c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 -- 172.21.15.132:0/2536982389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 msgr2=0x7fa26c1f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 0x7fa26c1f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa269e00640 1 -- 172.21.15.132:0/2536982389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c0c46f0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.349+0000 7fa26a800640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 0x7fa26c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa269e00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c500 tx=0x7fa25c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c017070 con 0x7fa26c06c8a0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25c005ce0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c0c49d0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c007330 con 0x7fa26c06c8a0 2024-03-27T06:19:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.350+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26c0c5290 con 0x7fa26c06c8a0 2024-03-27T06:19:42.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.351+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa26c0cbca0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.352+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa25c01a070 con 0x7fa26c06c8a0 2024-03-27T06:19:42.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.353+0000 7fa262000640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 0x7fa24c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:19:42.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.353+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fa25c08ad60 con 0x7fa26c06c8a0 2024-03-27T06:19:42.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.353+0000 7fa269400640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 0x7fa24c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:19:42.344 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.354+0000 7fa269400640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 0x7fa24c06e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa26c075a70 tx=0x7fa258009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:19:42.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.357+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa26c0cbca0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.603+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa26c068840 con 0x7fa26c06c8a0 2024-03-27T06:19:42.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.603+0000 7fa262000640 1 -- 172.21.15.132:0/2536982389 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7fa25c058bd0 con 0x7fa26c06c8a0 2024-03-27T06:19:42.594 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:19:42.595 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:19:42.595 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:19:42.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 msgr2=0x7fa24c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 0x7fa24c06e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa26c075a70 tx=0x7fa258009040 comp rx=0 tx=0).stop 2024-03-27T06:19:42.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 msgr2=0x7fa26c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:19:42.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c500 tx=0x7fa25c00c9c0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 shutdown_connections 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa26c0c8a30 0x7fa26c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa26c06cc80 0x7fa26c0c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa24c06c530 0x7fa24c06e9f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 --2- 172.21.15.132:0/2536982389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa26c06c8a0 0x7fa26c0c5a90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 >> 172.21.15.132:0/2536982389 conn(0x7fa26c0ccbe0 msgr2=0x7fa26c0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 shutdown_connections 2024-03-27T06:19:42.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:19:42.607+0000 7fa270c00640 1 -- 172.21.15.132:0/2536982389 wait complete. 2024-03-27T06:19:42.616 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-03-27T06:19:42.616 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2024-03-27T06:20:04.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:20:04.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:20:34.567 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:20:34.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:21:04.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:21:04.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:21:11.618 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:11.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.872+0000 7ff531800640 1 Processor -- start 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.872+0000 7ff531800640 1 -- start start 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff531800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 0x7ff52c0c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff531800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff52c06c990 0x7ff52c072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff531800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff52c0c80f0 con 0x7ff52c06c530 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff531800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff52c0b2080 con 0x7ff52c06c990 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff531800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff52c0c5cb0 con 0x7ff52c152740 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff52be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 0x7ff52c0c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff52b400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff52c06c530 0x7ff52c0b1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:57328/0 (socket says 172.21.15.132:57328) 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff52b400640 1 -- 172.21.15.132:0/4100327765 learned_addr learned my addr 172.21.15.132:0/4100327765 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.873+0000 7ff530e00640 1 --2- 172.21.15.132:0/4100327765 >> v2:172.21.15.132:3300/0 conn(0x7ff52c06c990 0x7ff52c072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52aa00640 1 -- 172.21.15.132:0/4100327765 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409258410 0 0) 0x7ff52c0c80f0 con 0x7ff52c06c530 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52aa00640 1 -- 172.21.15.132:0/4100327765 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff510003520 con 0x7ff52c06c530 2024-03-27T06:21:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52be00640 1 -- 172.21.15.132:0/4100327765 >> v2:172.21.15.132:3300/0 conn(0x7ff52c06c990 msgr2=0x7ff52c072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:11.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52be00640 1 --2- 172.21.15.132:0/4100327765 >> v2:172.21.15.132:3300/0 conn(0x7ff52c06c990 0x7ff52c072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52be00640 1 -- 172.21.15.132:0/4100327765 >> v1:172.21.15.134:6789/0 conn(0x7ff52c06c530 legacy=0x7ff52c0b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:11.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.874+0000 7ff52be00640 1 -- 172.21.15.132:0/4100327765 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c0c5f90 con 0x7ff52c152740 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.875+0000 7ff52be00640 1 --2- 172.21.15.132:0/4100327765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 0x7ff52c0c5640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0097a0 tx=0x7ff51c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4844a4a20417cd8e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.875+0000 7ff52aa00640 1 -- 172.21.15.132:0/4100327765 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c035020 con 0x7ff52c152740 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.875+0000 7ff52aa00640 1 -- 172.21.15.132:0/4100327765 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51c031810 con 0x7ff52c152740 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.875+0000 7ff52aa00640 1 -- 172.21.15.132:0/4100327765 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c031be0 con 0x7ff52c152740 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 -- 172.21.15.132:0/4100327765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 msgr2=0x7ff52c0c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 --2- 172.21.15.132:0/4100327765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 0x7ff52c0c5640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0097a0 tx=0x7ff51c02b8b0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 -- 172.21.15.132:0/4100327765 shutdown_connections 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 --2- 172.21.15.132:0/4100327765 >> v2:172.21.15.132:3300/0 conn(0x7ff52c06c990 0x7ff52c072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 --2- 172.21.15.132:0/4100327765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c152740 0x7ff52c0c5640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 -- 172.21.15.132:0/4100327765 >> 172.21.15.132:0/4100327765 conn(0x7ff52c0cc7a0 msgr2=0x7ff52c0c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 -- 172.21.15.132:0/4100327765 shutdown_connections 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.876+0000 7ff531800640 1 -- 172.21.15.132:0/4100327765 wait complete. 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.878+0000 7ff531800640 1 Processor -- start 2024-03-27T06:21:11.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.878+0000 7ff531800640 1 -- start start 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 0x7ff52c151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 0x7ff52c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c158db0 con 0x7ff52c152740 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c158ad0 con 0x7ff52c06c990 2024-03-27T06:21:11.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff531800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff52c152b20 con 0x7ff52c06c530 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:51282/0 (socket says 172.21.15.132:51282) 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 0x7ff52c151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 -- 172.21.15.132:0/4103415243 learned_addr learned my addr 172.21.15.132:0/4103415243 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff530e00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 0x7ff52c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 -- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 msgr2=0x7ff52c151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 0x7ff52c151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 -- 172.21.15.132:0/4103415243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 msgr2=0x7ff52c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 0x7ff52c0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff52be00640 1 -- 172.21.15.132:0/4103415243 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c150250 con 0x7ff52c06c530 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.879+0000 7ff530e00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 0x7ff52c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:21:11.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff52b400640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 0x7ff52c151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff52be00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0098c0 tx=0x7ff51c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c035040 con 0x7ff52c06c530 2024-03-27T06:21:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51c039cf0 con 0x7ff52c06c530 2024-03-27T06:21:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c039ea0 con 0x7ff52c06c530 2024-03-27T06:21:11.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c0c1920 con 0x7ff52c06c530 2024-03-27T06:21:11.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.880+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c150510 con 0x7ff52c06c530 2024-03-27T06:21:11.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.881+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c0686c0 con 0x7ff52c06c530 2024-03-27T06:21:11.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.882+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff51c038020 con 0x7ff52c06c530 2024-03-27T06:21:11.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.883+0000 7ff523e00640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 0x7ff50c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:11.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.883+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7ff51c0ad580 con 0x7ff52c06c530 2024-03-27T06:21:11.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.883+0000 7ff52b400640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 0x7ff50c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:11.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.884+0000 7ff52b400640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 0x7ff50c06e9f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0cb650 tx=0x7ff518009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:11.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:11.888+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7ff51c07b980 con 0x7ff52c06c530 2024-03-27T06:21:12.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.142+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff52c0b1e60 con 0x7ff52c06c530 2024-03-27T06:21:12.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.143+0000 7ff523e00640 1 -- 172.21.15.132:0/4103415243 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7ff51c07b280 con 0x7ff52c06c530 2024-03-27T06:21:12.133 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:12.133 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:12.133 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 msgr2=0x7ff50c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 0x7ff50c06e9f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0cb650 tx=0x7ff518009040 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 msgr2=0x7ff52c1f0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0098c0 tx=0x7ff51c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 shutdown_connections 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c152740 0x7ff52c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff50c06c530 0x7ff50c06e9f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff52c06c990 0x7ff52c151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 --2- 172.21.15.132:0/4103415243 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff52c06c530 0x7ff52c1f0db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 >> 172.21.15.132:0/4103415243 conn(0x7ff52c0cc7a0 msgr2=0x7ff52c074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:12.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 shutdown_connections 2024-03-27T06:21:12.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.146+0000 7ff531800640 1 -- 172.21.15.132:0/4103415243 wait complete. 2024-03-27T06:21:12.156 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2024-03-27T06:21:12.156 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-27T06:21:12.156 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:21:12.187 INFO:tasks.ceph.mds.c:Stopped 2024-03-27T06:21:12.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:12.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.447+0000 7fc4b3a00640 1 Processor -- start 2024-03-27T06:21:12.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.447+0000 7fc4b3a00640 1 -- start start 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.448+0000 7fc4b3a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4ac152950 0x7fc4ac1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.448+0000 7fc4b3a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4b3a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ac06b630 con 0x7fc4ac073970 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4b3a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fc4ac152d30 con 0x7fc4ac152950 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4b3a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac06b8d0 con 0x7fc4ac073550 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4b1000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4ac152950 0x7fc4ac1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:51298/0 (socket says 172.21.15.132:51298) 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 -- 172.21.15.132:0/28400361 learned_addr learned my addr 172.21.15.132:0/28400361 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4b1a00640 1 --1- 172.21.15.132:0/28400361 >> v1:172.21.15.134:6789/0 conn(0x7fc4ac073970 0x7fc4ac06cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:57340/0 (socket says 172.21.15.132:57340) 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 -- 172.21.15.132:0/28400361 >> v2:172.21.15.134:3300/0 conn(0x7fc4ac152950 msgr2=0x7fc4ac1590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 --2- 172.21.15.132:0/28400361 >> v2:172.21.15.134:3300/0 conn(0x7fc4ac152950 0x7fc4ac1590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 -- 172.21.15.132:0/28400361 >> v1:172.21.15.134:6789/0 conn(0x7fc4ac073970 legacy=0x7fc4ac06cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.449+0000 7fc4a3e00640 1 -- 172.21.15.132:0/28400361 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac06bbb0 con 0x7fc4ac073550 2024-03-27T06:21:12.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.450+0000 7fc4a3e00640 1 --2- 172.21.15.132:0/28400361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=980bce0d7ddcdfe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:12.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.450+0000 7fc4abe00640 1 -- 172.21.15.132:0/28400361 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494034070 con 0x7fc4ac073550 2024-03-27T06:21:12.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4abe00640 1 -- 172.21.15.132:0/28400361 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc494030840 con 0x7fc4ac073550 2024-03-27T06:21:12.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4abe00640 1 -- 172.21.15.132:0/28400361 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494030c10 con 0x7fc4ac073550 2024-03-27T06:21:12.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 -- 172.21.15.132:0/28400361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 msgr2=0x7fc4ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/28400361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49402b980 comp rx=0 tx=0).stop 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 -- 172.21.15.132:0/28400361 shutdown_connections 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/28400361 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac073550 0x7fc4ac071170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/28400361 >> v2:172.21.15.134:3300/0 conn(0x7fc4ac152950 0x7fc4ac1590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 -- 172.21.15.132:0/28400361 >> 172.21.15.132:0/28400361 conn(0x7fc4ac0cc060 msgr2=0x7fc4ac0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 -- 172.21.15.132:0/28400361 shutdown_connections 2024-03-27T06:21:12.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.451+0000 7fc4b3a00640 1 -- 172.21.15.132:0/28400361 wait complete. 2024-03-27T06:21:12.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.454+0000 7fc4b3a00640 1 Processor -- start 2024-03-27T06:21:12.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.454+0000 7fc4b3a00640 1 -- start start 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4ac073550 0x7fc4ac1f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 0x7fc4ac1eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 0x7fc4ac0cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac06c450 con 0x7fc4ac073550 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac06c170 con 0x7fc4ac073970 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b3a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac06d660 con 0x7fc4ac152950 2024-03-27T06:21:12.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4ac073550 0x7fc4ac1f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 0x7fc4ac0cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 0x7fc4ac1eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 0x7fc4ac1eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36120/0 (socket says 172.21.15.132:36120) 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4a3e00640 1 -- 172.21.15.132:0/838008943 learned_addr learned my addr 172.21.15.132:0/838008943 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 -- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 msgr2=0x7fc4ac1eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 0x7fc4ac1eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 -- 172.21.15.132:0/838008943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4ac073550 msgr2=0x7fc4ac1f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4ac073550 0x7fc4ac1f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.456+0000 7fc4b1a00640 1 -- 172.21.15.132:0/838008943 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac1eefb0 con 0x7fc4ac152950 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4b1a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 0x7fc4ac0cb410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c3f0 tx=0x7fc4a4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40074e0 con 0x7fc4ac152950 2024-03-27T06:21:12.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4a4007680 con 0x7fc4ac152950 2024-03-27T06:21:12.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac0c50b0 con 0x7fc4ac152950 2024-03-27T06:21:12.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40049e0 con 0x7fc4ac152950 2024-03-27T06:21:12.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.457+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac0c5670 con 0x7fc4ac152950 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.458+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ac0c58b0 con 0x7fc4ac152950 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.459+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc4a401a020 con 0x7fc4ac152950 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.460+0000 7fc4a9600640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 0x7fc48806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.460+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fc4a408af90 con 0x7fc4ac152950 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.460+0000 7fc4b1000640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 0x7fc48806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:12.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.461+0000 7fc4b1000640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 0x7fc48806e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc49c011340 tx=0x7fc49c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:12.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.467+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc4a4058cb0 con 0x7fc4ac152950 2024-03-27T06:21:12.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.732+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4ac068840 con 0x7fc4ac152950 2024-03-27T06:21:12.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.733+0000 7fc4a9600640 1 -- 172.21.15.132:0/838008943 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7fc4a405cfb0 con 0x7fc4ac152950 2024-03-27T06:21:12.722 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:12.722 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:12.723 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.735+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 msgr2=0x7fc48806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.735+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 0x7fc48806e9f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc49c011340 tx=0x7fc49c00b040 comp rx=0 tx=0).stop 2024-03-27T06:21:12.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.735+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 msgr2=0x7fc4ac0cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.735+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 0x7fc4ac0cb410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c3f0 tx=0x7fc4a4001e70 comp rx=0 tx=0).stop 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 shutdown_connections 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc4ac152950 0x7fc4ac0cb410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc48806c530 0x7fc48806e9f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ac073970 0x7fc4ac1eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 --2- 172.21.15.132:0/838008943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4ac073550 0x7fc4ac1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 >> 172.21.15.132:0/838008943 conn(0x7fc4ac0cc060 msgr2=0x7fc4ac1509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 shutdown_connections 2024-03-27T06:21:12.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:12.736+0000 7fc4b3a00640 1 -- 172.21.15.132:0/838008943 wait complete. 2024-03-27T06:21:12.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:14.745 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:14.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.956+0000 7fa0bfe00640 1 Processor -- start 2024-03-27T06:21:14.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.957+0000 7fa0bfe00640 1 -- start start 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.958+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bfe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa0c006be10 0x7fa0c015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bfe00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c00cdaa0 con 0x7fa0c01549a0 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bfe00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c00cada0 con 0x7fa0c006be10 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bfe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c015de10 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bea00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa0c01549a0 0x7fa0c00cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:57352/0 (socket says 172.21.15.132:57352) 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bea00640 1 -- 172.21.15.132:0/4254483796 learned_addr learned my addr 172.21.15.132:0/4254483796 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0b6000640 1 --2- 172.21.15.132:0/4254483796 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0bf400640 1 --2- 172.21.15.132:0/4254483796 >> v2:172.21.15.132:3300/0 conn(0x7fa0c006be10 0x7fa0c015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0b6000640 1 -- 172.21.15.132:0/4254483796 >> v2:172.21.15.132:3300/0 conn(0x7fa0c006be10 msgr2=0x7fa0c015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:14.949 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0b6000640 1 --2- 172.21.15.132:0/4254483796 >> v2:172.21.15.132:3300/0 conn(0x7fa0c006be10 0x7fa0c015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:14.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0be000640 1 -- 172.21.15.132:0/4254483796 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270712020 0 0) 0x7fa0b0001110 con 0x7fa0c01549a0 2024-03-27T06:21:14.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0b6000640 1 -- 172.21.15.132:0/4254483796 >> v1:172.21.15.134:6789/0 conn(0x7fa0c01549a0 legacy=0x7fa0c00cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:14.950 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.959+0000 7fa0b6000640 1 -- 172.21.15.132:0/4254483796 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c015e0b0 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.960+0000 7fa0b6000640 1 --2- 172.21.15.132:0/4254483796 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c006b8d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80097a0 tx=0x7fa0a802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771b7a9efdb29161 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0be000640 1 -- 172.21.15.132:0/4254483796 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8034070 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0be000640 1 -- 172.21.15.132:0/4254483796 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a8030800 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0be000640 1 -- 172.21.15.132:0/4254483796 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8030bd0 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 -- 172.21.15.132:0/4254483796 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 msgr2=0x7fa0c006b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/4254483796 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c006b8d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80097a0 tx=0x7fa0a802b940 comp rx=0 tx=0).stop 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 -- 172.21.15.132:0/4254483796 shutdown_connections 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/4254483796 >> v2:172.21.15.132:3300/0 conn(0x7fa0c006be10 0x7fa0c015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:14.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/4254483796 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c006b8d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:14.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 -- 172.21.15.132:0/4254483796 >> 172.21.15.132:0/4254483796 conn(0x7fa0c006d8d0 msgr2=0x7fa0c006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:14.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 -- 172.21.15.132:0/4254483796 shutdown_connections 2024-03-27T06:21:14.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.961+0000 7fa0bfe00640 1 -- 172.21.15.132:0/4254483796 wait complete. 2024-03-27T06:21:14.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.963+0000 7fa0bfe00640 1 Processor -- start 2024-03-27T06:21:14.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.963+0000 7fa0bfe00640 1 -- start start 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0c006be10 0x7fa0c01f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c00cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c015e950 con 0x7fa0c006be10 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c015e670 con 0x7fa0c01549a0 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bfe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c015ec30 con 0x7fa0c00cd6a0 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0c006be10 0x7fa0c01f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0b6000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c00cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36128/0 (socket says 172.21.15.132:36128) 2024-03-27T06:21:14.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0b6000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c00cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:51332/0 (socket says 172.21.15.132:51332) 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.964+0000 7fa0bf400640 1 -- 172.21.15.132:0/3017827510 learned_addr learned my addr 172.21.15.132:0/3017827510 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 -- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 msgr2=0x7fa0c00cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c00cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 -- 172.21.15.132:0/3017827510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0c006be10 msgr2=0x7fa0c01f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0c006be10 0x7fa0c01f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 -- 172.21.15.132:0/3017827510 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c0155610 con 0x7fa0c01549a0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bf400640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8007e60 tx=0x7fa0b800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8014420 con 0x7fa0c01549a0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b8019070 con 0x7fa0c01549a0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8014820 con 0x7fa0c01549a0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c00cfb60 con 0x7fa0c01549a0 2024-03-27T06:21:14.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.965+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c00d0120 con 0x7fa0c01549a0 2024-03-27T06:21:14.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.967+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c00cabc0 con 0x7fa0c01549a0 2024-03-27T06:21:14.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.967+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa0b80149c0 con 0x7fa0c01549a0 2024-03-27T06:21:14.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.968+0000 7fa0b7400640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 0x7fa09c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:14.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.968+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fa0b808bff0 con 0x7fa0c01549a0 2024-03-27T06:21:14.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.969+0000 7fa0bea00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 0x7fa09c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:14.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.969+0000 7fa0bea00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 0x7fa09c06ea20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0000f60 tx=0x7fa0b0007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:14.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:14.973+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa0c00cabc0 con 0x7fa0c01549a0 2024-03-27T06:21:15.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.212+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0c00686c0 con 0x7fa0c01549a0 2024-03-27T06:21:15.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.213+0000 7fa0b7400640 1 -- 172.21.15.132:0/3017827510 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7fa0b8059d70 con 0x7fa0c01549a0 2024-03-27T06:21:15.202 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:15.202 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:15.203 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 msgr2=0x7fa09c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 0x7fa09c06ea20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0000f60 tx=0x7fa0b0007b50 comp rx=0 tx=0).stop 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 msgr2=0x7fa0c0156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8007e60 tx=0x7fa0b800e4b0 comp rx=0 tx=0).stop 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 shutdown_connections 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa09c06c560 0x7fa09c06ea20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c01549a0 0x7fa0c0156f10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:15.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa0c00cd6a0 0x7fa0c00cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:15.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 --2- 172.21.15.132:0/3017827510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0c006be10 0x7fa0c01f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:15.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 >> 172.21.15.132:0/3017827510 conn(0x7fa0c006d8d0 msgr2=0x7fa0c0158500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:15.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 shutdown_connections 2024-03-27T06:21:15.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:15.215+0000 7fa0bfe00640 1 -- 172.21.15.132:0/3017827510 wait complete. 2024-03-27T06:21:15.224 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:17.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:17.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.479+0000 7f5bff200640 1 Processor -- start 2024-03-27T06:21:17.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.479+0000 7f5bff200640 1 -- start start 2024-03-27T06:21:17.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.480+0000 7f5bff200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:17.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bff200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0015de10 con 0x7f5c000cb870 2024-03-27T06:21:17.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bff200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0015e0b0 con 0x7f5c0015adb0 2024-03-27T06:21:17.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bff200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5c000cada0 con 0x7f5c001549a0 2024-03-27T06:21:17.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:17.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:51346/0 (socket says 172.21.15.132:51346) 2024-03-27T06:21:17.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 -- 172.21.15.132:0/78431775 learned_addr learned my addr 172.21.15.132:0/78431775 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:17.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfe800640 1 --1- 172.21.15.132:0/78431775 >> v1:172.21.15.134:6789/0 conn(0x7f5c0015adb0 0x7f5c0015b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:57358/0 (socket says 172.21.15.132:57358) 2024-03-27T06:21:17.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 -- 172.21.15.132:0/78431775 >> v1:172.21.15.134:6789/0 conn(0x7f5c0015adb0 legacy=0x7f5c0015b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 -- 172.21.15.132:0/78431775 >> v1:172.21.15.132:6789/0 conn(0x7f5c000cb870 legacy=0x7f5c0006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfde00640 1 -- 172.21.15.132:0/78431775 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0015e390 con 0x7f5c001549a0 2024-03-27T06:21:17.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.481+0000 7f5bfca00640 1 -- 172.21.15.132:0/78431775 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819605378 0 0) 0x7f5c0015de10 con 0x7f5c000cb870 2024-03-27T06:21:17.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.483+0000 7f5bfde00640 1 --2- 172.21.15.132:0/78431775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00098a0 tx=0x7f5bf002baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bbdb14d769243c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:17.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.483+0000 7f5bfca00640 1 -- 172.21.15.132:0/78431775 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002d860 con 0x7f5c001549a0 2024-03-27T06:21:17.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.483+0000 7f5bfca00640 1 -- 172.21.15.132:0/78431775 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf002da00 con 0x7f5c001549a0 2024-03-27T06:21:17.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.483+0000 7f5bfca00640 1 -- 172.21.15.132:0/78431775 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002ddd0 con 0x7f5c001549a0 2024-03-27T06:21:17.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 -- 172.21.15.132:0/78431775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 msgr2=0x7f5c000cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 --2- 172.21.15.132:0/78431775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00098a0 tx=0x7f5bf002baa0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 -- 172.21.15.132:0/78431775 shutdown_connections 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 --2- 172.21.15.132:0/78431775 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c001549a0 0x7f5c000cd6a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 -- 172.21.15.132:0/78431775 >> 172.21.15.132:0/78431775 conn(0x7f5c0006d8d0 msgr2=0x7f5c0006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 -- 172.21.15.132:0/78431775 shutdown_connections 2024-03-27T06:21:17.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.484+0000 7f5bff200640 1 -- 172.21.15.132:0/78431775 wait complete. 2024-03-27T06:21:17.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.485+0000 7f5bff200640 1 Processor -- start 2024-03-27T06:21:17.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.486+0000 7f5bff200640 1 -- start start 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.486+0000 7f5bff200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.486+0000 7f5bff200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.486+0000 7f5bff200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 0x7f5c000c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bff200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0015ec30 con 0x7f5c000cb870 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bff200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0015e950 con 0x7f5c001549a0 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bff200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0015a820 con 0x7f5c0015adb0 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfe800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 0x7f5c000c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36136/0 (socket says 172.21.15.132:36136) 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53588/0 (socket says 172.21.15.132:53588) 2024-03-27T06:21:17.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfd400640 1 -- 172.21.15.132:0/3833489002 learned_addr learned my addr 172.21.15.132:0/3833489002 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 -- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 msgr2=0x7f5c000c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 0x7f5c000c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 -- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 msgr2=0x7f5c000c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfde00640 1 -- 172.21.15.132:0/3833489002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c000ce650 con 0x7f5c000cb870 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfd400640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:17.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.487+0000 7f5bfe800640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 0x7f5c000c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bfde00640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c0015dfe0 tx=0x7f5bf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0047070 con 0x7f5c000cb870 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf0037ac0 con 0x7f5c000cb870 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002e430 con 0x7f5c000cb870 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c000c8410 con 0x7f5c000cb870 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.488+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c000c89b0 con 0x7f5c000cb870 2024-03-27T06:21:17.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.489+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c00074e60 con 0x7f5c000cb870 2024-03-27T06:21:17.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.490+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5bf00423f0 con 0x7f5c000cb870 2024-03-27T06:21:17.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.491+0000 7f5bf6000640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 0x7f5be006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:17.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.491+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f5bf00ad970 con 0x7f5c000cb870 2024-03-27T06:21:17.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.492+0000 7f5bfd400640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 0x7f5be006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:17.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.492+0000 7f5bfd400640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 0x7f5be006ea20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c000b3a50 tx=0x7f5bec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:17.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.496+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5c00074e60 con 0x7f5c000cb870 2024-03-27T06:21:17.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.742+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c000686c0 con 0x7f5c000cb870 2024-03-27T06:21:17.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.743+0000 7f5bf6000640 1 -- 172.21.15.132:0/3833489002 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7f5c000ce650 con 0x7f5c000cb870 2024-03-27T06:21:17.733 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:17.733 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:17.733 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 msgr2=0x7f5be006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 0x7f5be006ea20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c000b3a50 tx=0x7f5bec009040 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 msgr2=0x7f5c001f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c0015dfe0 tx=0x7f5bf0004730 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 shutdown_connections 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c0015adb0 0x7f5c000c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5be006c560 0x7f5be006ea20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c001549a0 0x7f5c000c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 --2- 172.21.15.132:0/3833489002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c000cb870 0x7f5c001f43b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 >> 172.21.15.132:0/3833489002 conn(0x7f5c0006d8d0 msgr2=0x7f5c0015b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:17.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 shutdown_connections 2024-03-27T06:21:17.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:17.746+0000 7f5bff200640 1 -- 172.21.15.132:0/3833489002 wait complete. 2024-03-27T06:21:17.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:19.757 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.973+0000 7f46efe00640 1 Processor -- start 2024-03-27T06:21:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.973+0000 7f46efe00640 1 -- start start 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.975+0000 7f46efe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f46f00c7920 0x7f46f0071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.975+0000 7f46efe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 0x7f46f0157590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.975+0000 7f46efe00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f00737c0 con 0x7f46f0151660 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.975+0000 7f46efe00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f46f00732a0 con 0x7f46f00c7920 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.975+0000 7f46efe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f46f0073aa0 con 0x7f46f006d370 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46eea00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f46f0151660 0x7f46f006c630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:57372/0 (socket says 172.21.15.132:57372) 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46eea00640 1 -- 172.21.15.132:0/2545140131 learned_addr learned my addr 172.21.15.132:0/2545140131 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ef400640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 0x7f46f0157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ee000640 1 --2- 172.21.15.132:0/2545140131 >> v2:172.21.15.134:3300/0 conn(0x7f46f00c7920 0x7f46f0071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ed600640 1 -- 172.21.15.132:0/2545140131 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898326369 0 0) 0x7f46f00737c0 con 0x7f46f0151660 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ed600640 1 -- 172.21.15.132:0/2545140131 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46d4003520 con 0x7f46f0151660 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ee000640 1 -- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 msgr2=0x7f46f0157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ee000640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 0x7f46f0157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ee000640 1 -- 172.21.15.132:0/2545140131 >> v1:172.21.15.134:6789/0 conn(0x7f46f0151660 legacy=0x7f46f006c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ee000640 1 -- 172.21.15.132:0/2545140131 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0072c50 con 0x7f46f00c7920 2024-03-27T06:21:19.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.976+0000 7f46ef400640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 0x7f46f0157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:19.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.978+0000 7f46ee000640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f00c7920 0x7f46f0071f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f5871ddd1d6ec92 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:19.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.978+0000 7f46ed600640 1 -- 172.21.15.132:0/2545140131 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc035020 con 0x7f46f00c7920 2024-03-27T06:21:19.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.978+0000 7f46ed600640 1 -- 172.21.15.132:0/2545140131 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc031810 con 0x7f46f00c7920 2024-03-27T06:21:19.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.978+0000 7f46ed600640 1 -- 172.21.15.132:0/2545140131 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc031be0 con 0x7f46f00c7920 2024-03-27T06:21:19.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 -- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f00c7920 msgr2=0x7f46f0071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f00c7920 0x7f46f0071f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b8b0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 -- 172.21.15.132:0/2545140131 shutdown_connections 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f006d370 0x7f46f0157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 --2- 172.21.15.132:0/2545140131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f00c7920 0x7f46f0071f30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 -- 172.21.15.132:0/2545140131 >> 172.21.15.132:0/2545140131 conn(0x7f46f00c99b0 msgr2=0x7f46f00c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 -- 172.21.15.132:0/2545140131 shutdown_connections 2024-03-27T06:21:19.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.979+0000 7f46efe00640 1 -- 172.21.15.132:0/2545140131 wait complete. 2024-03-27T06:21:19.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.981+0000 7f46efe00640 1 Processor -- start 2024-03-27T06:21:19.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.981+0000 7f46efe00640 1 -- start start 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f006d370 0x7f46f00cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 0x7f46f014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46f006cd40 con 0x7f46f006d370 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f46f0151a40 con 0x7f46f0151660 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.982+0000 7f46efe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f46f006cec0 con 0x7f46f00c7920 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ef400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46eea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f006d370 0x7f46f00cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 0x7f46f014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ef400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36154/0 (socket says 172.21.15.132:36154) 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ef400640 1 -- 172.21.15.132:0/391125810 learned_addr learned my addr 172.21.15.132:0/391125810 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 -- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 msgr2=0x7f46f0149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:19.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 -- 172.21.15.132:0/391125810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f006d370 msgr2=0x7f46f00cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f006d370 0x7f46f00cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ee000640 1 -- 172.21.15.132:0/391125810 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f014cba0 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.983+0000 7f46ef400640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46ee000640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 0x7f46f014c660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f46dc033000 tx=0x7f46dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc041070 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc031e40 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc035040 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f014d8d0 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.984+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f014d5f0 con 0x7f46f00c7920 2024-03-27T06:21:19.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.985+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f00686c0 con 0x7f46f00c7920 2024-03-27T06:21:19.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.987+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f46dc04c060 con 0x7f46f00c7920 2024-03-27T06:21:19.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.988+0000 7f46e6a00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 0x7f46d006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:19.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.988+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f46dc0ad2d0 con 0x7f46f00c7920 2024-03-27T06:21:19.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.989+0000 7f46eea00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 0x7f46d006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:19.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.989+0000 7f46eea00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 0x7f46d006ea40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46e00011e0 tx=0x7f46e0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:19.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:19.997+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f46dc07b070 con 0x7f46f00c7920 2024-03-27T06:21:20.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.252+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f00c2070 con 0x7f46f00c7920 2024-03-27T06:21:20.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.253+0000 7f46e6a00640 1 -- 172.21.15.132:0/391125810 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7f46dc07f370 con 0x7f46f00c7920 2024-03-27T06:21:20.243 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:20.243 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:20.243 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:20.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 msgr2=0x7f46d006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:20.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 0x7f46d006ea40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46e00011e0 tx=0x7f46e0007d30 comp rx=0 tx=0).stop 2024-03-27T06:21:20.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 msgr2=0x7f46f014c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:20.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 0x7f46f014c660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f46dc033000 tx=0x7f46dc0046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 shutdown_connections 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f46d006c580 0x7f46d006ea40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46f0151660 0x7f46f0149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f46f00c7920 0x7f46f014c660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 --2- 172.21.15.132:0/391125810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f006d370 0x7f46f00cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 >> 172.21.15.132:0/391125810 conn(0x7f46f00c99b0 msgr2=0x7f46f014a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 shutdown_connections 2024-03-27T06:21:20.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:20.256+0000 7f46efe00640 1 -- 172.21.15.132:0/391125810 wait complete. 2024-03-27T06:21:20.267 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:22.269 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:22.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.490+0000 7fcd84200640 1 Processor -- start 2024-03-27T06:21:22.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.491+0000 7fcd84200640 1 -- start start 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.492+0000 7fcd84200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcd7c0c9150 0x7fcd7c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.492+0000 7fcd84200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7c0cb9f0 con 0x7fcd7c0d0970 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.492+0000 7fcd84200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7c15e2e0 con 0x7fcd7c0d0d50 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.492+0000 7fcd84200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fcd7c15e5c0 con 0x7fcd7c0c9150 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.492+0000 7fcd81800640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fcd7c0d0970 0x7fcd7c154aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37124/0 (socket says 172.21.15.132:37124) 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd81800640 1 -- 172.21.15.132:0/311951809 learned_addr learned my addr 172.21.15.132:0/311951809 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:22.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd82200640 1 --2- 172.21.15.132:0/311951809 >> v2:172.21.15.134:3300/0 conn(0x7fcd7c0c9150 0x7fcd7c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154992631 0 0) 0x7fcd7c15e2e0 con 0x7fcd7c0d0d50 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd64003520 con 0x7fcd7c0d0d50 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310564943 0 0) 0x7fcd7c0cb9f0 con 0x7fcd7c0d0970 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd82200640 1 -- 172.21.15.132:0/311951809 >> v1:172.21.15.134:6789/0 conn(0x7fcd7c0d0d50 legacy=0x7fcd7c0c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd82200640 1 -- 172.21.15.132:0/311951809 >> v1:172.21.15.132:6789/0 conn(0x7fcd7c0d0970 legacy=0x7fcd7c154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.493+0000 7fcd82200640 1 -- 172.21.15.132:0/311951809 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c15e8a0 con 0x7fcd7c0c9150 2024-03-27T06:21:22.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.494+0000 7fcd82200640 1 --2- 172.21.15.132:0/311951809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c15b6d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009d90 tx=0x7fcd7402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af5352bb516e9dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:22.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.494+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74039070 con 0x7fcd7c0c9150 2024-03-27T06:21:22.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.494+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd7402ccf0 con 0x7fcd7c0c9150 2024-03-27T06:21:22.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.494+0000 7fcd7be00640 1 -- 172.21.15.132:0/311951809 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74034660 con 0x7fcd7c0c9150 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 -- 172.21.15.132:0/311951809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 msgr2=0x7fcd7c15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 --2- 172.21.15.132:0/311951809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c15b6d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009d90 tx=0x7fcd7402bf40 comp rx=0 tx=0).stop 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 -- 172.21.15.132:0/311951809 shutdown_connections 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 --2- 172.21.15.132:0/311951809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c15b6d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 -- 172.21.15.132:0/311951809 >> 172.21.15.132:0/311951809 conn(0x7fcd7c06dbd0 msgr2=0x7fcd7c06dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 -- 172.21.15.132:0/311951809 shutdown_connections 2024-03-27T06:21:22.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.495+0000 7fcd84200640 1 -- 172.21.15.132:0/311951809 wait complete. 2024-03-27T06:21:22.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.497+0000 7fcd84200640 1 Processor -- start 2024-03-27T06:21:22.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.497+0000 7fcd84200640 1 -- start start 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd7c0d0970 0x7fcd7c14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 0x7fcd7c150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c155330 con 0x7fcd7c0d0970 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c1551b0 con 0x7fcd7c0c9150 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd84200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c0cc7f0 con 0x7fcd7c0d0d50 2024-03-27T06:21:22.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd81800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd82200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 0x7fcd7c150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd81800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:55940/0 (socket says 172.21.15.132:55940) 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd81800640 1 -- 172.21.15.132:0/748012726 learned_addr learned my addr 172.21.15.132:0/748012726 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.498+0000 7fcd80e00640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd7c0d0970 0x7fcd7c14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 -- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 msgr2=0x7fcd7c150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 0x7fcd7c150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 -- 172.21.15.132:0/748012726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd7c0d0970 msgr2=0x7fcd7c14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd7c0d0970 0x7fcd7c14c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 -- 172.21.15.132:0/748012726 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c158c30 con 0x7fcd7c0c9150 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd82200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 0x7fcd7c150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd81800640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000c4d0 tx=0x7fcd7000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70018070 con 0x7fcd7c0c9150 2024-03-27T06:21:22.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70007500 con 0x7fcd7c0c9150 2024-03-27T06:21:22.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70004a20 con 0x7fcd7c0c9150 2024-03-27T06:21:22.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c158f10 con 0x7fcd7c0c9150 2024-03-27T06:21:22.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.499+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c0cbc70 con 0x7fcd7c0c9150 2024-03-27T06:21:22.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.500+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7c0cb440 con 0x7fcd7c0c9150 2024-03-27T06:21:22.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.501+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fcd70004bc0 con 0x7fcd7c0c9150 2024-03-27T06:21:22.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.502+0000 7fcd79600640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 0x7fcd5c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:22.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.502+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fcd7008ab90 con 0x7fcd7c0c9150 2024-03-27T06:21:22.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.502+0000 7fcd80e00640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 0x7fcd5c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:22.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.503+0000 7fcd80e00640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 0x7fcd5c06ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd68007fd0 tx=0x7fcd68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:22.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.506+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fcd7c0cb440 con 0x7fcd7c0c9150 2024-03-27T06:21:22.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.753+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd7c0c7650 con 0x7fcd7c0c9150 2024-03-27T06:21:22.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.754+0000 7fcd79600640 1 -- 172.21.15.132:0/748012726 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7fcd7c0c7650 con 0x7fcd7c0c9150 2024-03-27T06:21:22.744 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:22.744 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:22.744 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:22.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 msgr2=0x7fcd5c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 0x7fcd5c06ea20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd68007fd0 tx=0x7fcd68009040 comp rx=0 tx=0).stop 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 msgr2=0x7fcd7c1f4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000c4d0 tx=0x7fcd7000c990 comp rx=0 tx=0).stop 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 shutdown_connections 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fcd7c0d0d50 0x7fcd7c150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd7c0d0970 0x7fcd7c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fcd5c06c560 0x7fcd5c06ea20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 --2- 172.21.15.132:0/748012726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcd7c0c9150 0x7fcd7c1f4380 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:22.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.757+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 >> 172.21.15.132:0/748012726 conn(0x7fcd7c06dbd0 msgr2=0x7fcd7c06dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:22.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.758+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 shutdown_connections 2024-03-27T06:21:22.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:22.758+0000 7fcd84200640 1 -- 172.21.15.132:0/748012726 wait complete. 2024-03-27T06:21:22.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:24.769 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:25.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.012+0000 7f756ca00640 1 Processor -- start 2024-03-27T06:21:25.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.012+0000 7f756ca00640 1 -- start start 2024-03-27T06:21:25.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f756ca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 0x7f75680c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:25.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f756ca00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75680c80f0 con 0x7f756806c990 2024-03-27T06:21:25.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f756ca00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75680c5cb0 con 0x7f756806c530 2024-03-27T06:21:25.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f756ca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f75680b2080 con 0x7f7568152740 2024-03-27T06:21:25.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f755ea00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f756806c530 0x7f75680b1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56468/0 (socket says 172.21.15.132:56468) 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f755ea00640 1 -- 172.21.15.132:0/3169715552 learned_addr learned my addr 172.21.15.132:0/3169715552 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f7567400640 1 --2- 172.21.15.132:0/3169715552 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 0x7f75680c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f7566a00640 1 -- 172.21.15.132:0/3169715552 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405651372 0 0) 0x7f75680c5cb0 con 0x7f756806c530 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.014+0000 7f7566a00640 1 -- 172.21.15.132:0/3169715552 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f754c003520 con 0x7f756806c530 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.015+0000 7f7567400640 1 -- 172.21.15.132:0/3169715552 >> v1:172.21.15.134:6789/0 conn(0x7f756806c530 legacy=0x7f75680b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.015+0000 7f7567400640 1 -- 172.21.15.132:0/3169715552 >> v1:172.21.15.132:6789/0 conn(0x7f756806c990 legacy=0x7f7568072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.015+0000 7f7567400640 1 -- 172.21.15.132:0/3169715552 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75680c5f90 con 0x7f7568152740 2024-03-27T06:21:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f7567400640 1 --2- 172.21.15.132:0/3169715552 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 0x7f75680c5640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7558009dc0 tx=0x7f755802bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4776836dcd84560d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f7566a00640 1 -- 172.21.15.132:0/3169715552 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558039070 con 0x7f7568152740 2024-03-27T06:21:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f7566a00640 1 -- 172.21.15.132:0/3169715552 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f755802ccc0 con 0x7f7568152740 2024-03-27T06:21:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f7566a00640 1 -- 172.21.15.132:0/3169715552 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558034660 con 0x7f7568152740 2024-03-27T06:21:25.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f756ca00640 1 -- 172.21.15.132:0/3169715552 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 msgr2=0x7f75680c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.016+0000 7f756ca00640 1 --2- 172.21.15.132:0/3169715552 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 0x7f75680c5640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7558009dc0 tx=0x7f755802bf10 comp rx=0 tx=0).stop 2024-03-27T06:21:25.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.017+0000 7f756ca00640 1 -- 172.21.15.132:0/3169715552 shutdown_connections 2024-03-27T06:21:25.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.017+0000 7f756ca00640 1 --2- 172.21.15.132:0/3169715552 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7568152740 0x7f75680c5640 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.017+0000 7f756ca00640 1 -- 172.21.15.132:0/3169715552 >> 172.21.15.132:0/3169715552 conn(0x7f75680cc7a0 msgr2=0x7f75680c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:25.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.017+0000 7f756ca00640 1 -- 172.21.15.132:0/3169715552 shutdown_connections 2024-03-27T06:21:25.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.017+0000 7f756ca00640 1 -- 172.21.15.132:0/3169715552 wait complete. 2024-03-27T06:21:25.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.018+0000 7f756ca00640 1 Processor -- start 2024-03-27T06:21:25.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.019+0000 7f756ca00640 1 -- start start 2024-03-27T06:21:25.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.019+0000 7f756ca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:25.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f756ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f756806c990 0x7f7568070de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:25.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f756ca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 0x7f756814ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:25.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f756ca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7568158db0 con 0x7f7568152740 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f756ca00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7568158ad0 con 0x7f756806c990 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f756ca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7568152b20 con 0x7f756806c530 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:39224/0 (socket says 172.21.15.132:39224) 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 -- 172.21.15.132:0/3583616841 learned_addr learned my addr 172.21.15.132:0/3583616841 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:25.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f755ea00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f756806c990 0x7f7568070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:25.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567e00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 0x7f756814ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:25.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 -- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f756806c990 msgr2=0x7f7568070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f756806c990 0x7f7568070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 -- 172.21.15.132:0/3583616841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 msgr2=0x7f756814ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 0x7f756814ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 -- 172.21.15.132:0/3583616841 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75680c9fa0 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567e00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 0x7f756814ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.020+0000 7f7567400640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7558037040 tx=0x7f7558004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.021+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558039050 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.021+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7558045070 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.021+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75680ca280 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.021+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755803e850 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.021+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75681f0360 con 0x7f756806c530 2024-03-27T06:21:25.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.022+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75680686c0 con 0x7f756806c530 2024-03-27T06:21:25.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.023+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f755803e9f0 con 0x7f756806c530 2024-03-27T06:21:25.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.024+0000 7f755fe00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 0x7f754406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:25.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.024+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7f75580ae660 con 0x7f756806c530 2024-03-27T06:21:25.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.024+0000 7f755ea00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 0x7f754406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:25.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.025+0000 7f755ea00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 0x7f754406e9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f756814a760 tx=0x7f7550007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:25.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.031+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f755807caf0 con 0x7f756806c530 2024-03-27T06:21:25.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.404+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f756806c120 con 0x7f756806c530 2024-03-27T06:21:25.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.405+0000 7f755fe00640 1 -- 172.21.15.132:0/3583616841 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7f755807c3f0 con 0x7f756806c530 2024-03-27T06:21:25.395 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:25.396 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:25.396 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:25.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 msgr2=0x7f754406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 0x7f754406e9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f756814a760 tx=0x7f7550007d30 comp rx=0 tx=0).stop 2024-03-27T06:21:25.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 msgr2=0x7f75681f0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7558037040 tx=0x7f7558004a40 comp rx=0 tx=0).stop 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 shutdown_connections 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7568152740 0x7f756814ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f754406c530 0x7f754406e9f0 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f756814a760 tx=0x7f7550007d30 comp rx=0 tx=0).stop 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f756806c990 0x7f7568070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 --2- 172.21.15.132:0/3583616841 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f756806c530 0x7f75681f0e40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:25.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.415+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 >> 172.21.15.132:0/3583616841 conn(0x7f75680cc7a0 msgr2=0x7f756814d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:25.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.417+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 shutdown_connections 2024-03-27T06:21:25.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:25.417+0000 7f756ca00640 1 -- 172.21.15.132:0/3583616841 wait complete. 2024-03-27T06:21:25.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:27.446 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:27.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.680+0000 7fddde000640 1 Processor -- start 2024-03-27T06:21:27.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.680+0000 7fddde000640 1 -- start start 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.681+0000 7fddde000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fddd80b0d10 0x7fddd806fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.681+0000 7fddde000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd806f6c0 con 0x7fddd8159c30 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.681+0000 7fddde000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd80c3760 con 0x7fddd8152c60 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddde000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fddd806f250 con 0x7fddd80b0d10 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fddd80b0d10 0x7fddd806fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fddd8152c60 0x7fddd80c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56478/0 (socket says 172.21.15.132:56478) 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7400640 1 -- 172.21.15.132:0/2226649061 learned_addr learned my addr 172.21.15.132:0/2226649061 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd6a00640 1 --1- 172.21.15.132:0/2226649061 >> v1:172.21.15.132:6789/0 conn(0x7fddd8159c30 0x7fddd80ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37130/0 (socket says 172.21.15.132:37130) 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7e00640 1 -- 172.21.15.132:0/2226649061 >> v1:172.21.15.134:6789/0 conn(0x7fddd8152c60 legacy=0x7fddd80c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7e00640 1 -- 172.21.15.132:0/2226649061 >> v1:172.21.15.132:6789/0 conn(0x7fddd8159c30 legacy=0x7fddd80ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd6000640 1 -- 172.21.15.132:0/2226649061 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196512241 0 0) 0x7fddd80c3760 con 0x7fddd8152c60 2024-03-27T06:21:27.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.682+0000 7fddd7e00640 1 -- 172.21.15.132:0/2226649061 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd815c7e0 con 0x7fddd80b0d10 2024-03-27T06:21:27.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.684+0000 7fddd7e00640 1 --2- 172.21.15.132:0/2226649061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80b0d10 0x7fddd806fb10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009d90 tx=0x7fddd002bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=257ff9bc53ca0e1b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:27.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.684+0000 7fddd6000640 1 -- 172.21.15.132:0/2226649061 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0039070 con 0x7fddd80b0d10 2024-03-27T06:21:27.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.684+0000 7fddd6000640 1 -- 172.21.15.132:0/2226649061 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd002ccf0 con 0x7fddd80b0d10 2024-03-27T06:21:27.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.684+0000 7fddd6000640 1 -- 172.21.15.132:0/2226649061 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0034660 con 0x7fddd80b0d10 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 -- 172.21.15.132:0/2226649061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80b0d10 msgr2=0x7fddd806fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 --2- 172.21.15.132:0/2226649061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80b0d10 0x7fddd806fb10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009d90 tx=0x7fddd002bf40 comp rx=0 tx=0).stop 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 -- 172.21.15.132:0/2226649061 shutdown_connections 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 --2- 172.21.15.132:0/2226649061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80b0d10 0x7fddd806fb10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 -- 172.21.15.132:0/2226649061 >> 172.21.15.132:0/2226649061 conn(0x7fddd8067280 msgr2=0x7fddd8067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 -- 172.21.15.132:0/2226649061 shutdown_connections 2024-03-27T06:21:27.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.685+0000 7fddde000640 1 -- 172.21.15.132:0/2226649061 wait complete. 2024-03-27T06:21:27.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.687+0000 7fddde000640 1 Processor -- start 2024-03-27T06:21:27.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.687+0000 7fddde000640 1 -- start start 2024-03-27T06:21:27.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 0x7fddd81e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 0x7fddd81e1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80ce9a0 con 0x7fddd80b0d10 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fddd806eee0 con 0x7fddd8159c30 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.688+0000 7fddde000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80ceb20 con 0x7fddd8152c60 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 0x7fddd81e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 0x7fddd81e1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57802/0 (socket says 172.21.15.132:57802) 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7400640 1 -- 172.21.15.132:0/2077348229 learned_addr learned my addr 172.21.15.132:0/2077348229 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 -- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 msgr2=0x7fddd81e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 0x7fddd81e1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 -- 172.21.15.132:0/2077348229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 msgr2=0x7fddd81f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd6a00640 1 -- 172.21.15.132:0/2077348229 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd81e44b0 con 0x7fddd8152c60 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7e00640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 0x7fddd81e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.689+0000 7fddd7400640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddd6a00640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 0x7fddd81e38f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fddc400c3f0 tx=0x7fddc400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc40074e0 con 0x7fddd8152c60 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc4007680 con 0x7fddd8152c60 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc40049e0 con 0x7fddd8152c60 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd81f4430 con 0x7fddd8152c60 2024-03-27T06:21:27.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.690+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd80cabe0 con 0x7fddd8152c60 2024-03-27T06:21:27.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.691+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8068840 con 0x7fddd8152c60 2024-03-27T06:21:27.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.693+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fddc401a020 con 0x7fddd8152c60 2024-03-27T06:21:27.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.694+0000 7fddcf400640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 0x7fddb806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:27.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.694+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6082+0+0 (secure 0 0 0) 0x7fddc408ae10 con 0x7fddd8152c60 2024-03-27T06:21:27.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.694+0000 7fddd7400640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 0x7fddb806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:27.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.695+0000 7fddd7400640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 0x7fddb806e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007b10 tx=0x7fddc8007a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:27.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.699+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fddc408e030 con 0x7fddd8152c60 2024-03-27T06:21:27.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.938+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd815af70 con 0x7fddd8152c60 2024-03-27T06:21:27.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.938+0000 7fddcf400640 1 -- 172.21.15.132:0/2077348229 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6890 (secure 0 0 0) 0x7fddc4058c50 con 0x7fddd8152c60 2024-03-27T06:21:27.928 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:27.928 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5284,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:19:33.509129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":4718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4718":{"gid":4718,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":21,"addr":"172.21.15.132:6837/1424864911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1424864911},{"type":"v1","addr":"172.21.15.132:6837","nonce":1424864911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817,4718]},"id":1}]} 2024-03-27T06:21:27.928 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 36 2024-03-27T06:21:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 msgr2=0x7fddb806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 0x7fddb806e9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007b10 tx=0x7fddc8007a60 comp rx=0 tx=0).stop 2024-03-27T06:21:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 msgr2=0x7fddd81e38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 0x7fddd81e38f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fddc400c3f0 tx=0x7fddc400c8b0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 shutdown_connections 2024-03-27T06:21:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fddb806c530 0x7fddb806e9f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd8159c30 0x7fddd81e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fddd8152c60 0x7fddd81e38f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 --2- 172.21.15.132:0/2077348229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fddd80b0d10 0x7fddd81f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 >> 172.21.15.132:0/2077348229 conn(0x7fddd8067280 msgr2=0x7fddd815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 shutdown_connections 2024-03-27T06:21:27.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:27.941+0000 7fddde000640 1 -- 172.21.15.132:0/2077348229 wait complete. 2024-03-27T06:21:27.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-27T06:21:29.954 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:30.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.176+0000 7f3630400640 1 Processor -- start 2024-03-27T06:21:30.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.176+0000 7f3630400640 1 -- start start 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3630400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f36281516f0 0x7f36280c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3630400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36280c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3630400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f362806b520 con 0x7f3628151ad0 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3630400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f3628072be0 con 0x7f36281516f0 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3630400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f362806b7c0 con 0x7f3628157c80 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f362da00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f36281516f0 0x7f36280c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f362d000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3628151ad0 0x7f3628157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56490/0 (socket says 172.21.15.132:56490) 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f362d000640 1 -- 172.21.15.132:0/3823680454 learned_addr learned my addr 172.21.15.132:0/3823680454 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f362e400640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36280c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3627e00640 1 -- 172.21.15.132:0/3823680454 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310004758 0 0) 0x7f362806b520 con 0x7f3628151ad0 2024-03-27T06:21:30.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.178+0000 7f3627e00640 1 -- 172.21.15.132:0/3823680454 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3610003520 con 0x7f3628151ad0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.179+0000 7f362da00640 1 -- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 msgr2=0x7f36280c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.179+0000 7f362da00640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36280c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.179+0000 7f362da00640 1 -- 172.21.15.132:0/3823680454 >> v1:172.21.15.134:6789/0 conn(0x7f3628151ad0 legacy=0x7f3628157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.179+0000 7f362da00640 1 -- 172.21.15.132:0/3823680454 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362806baa0 con 0x7f36281516f0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.180+0000 7f362da00640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36280c7e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f361c0097a0 tx=0x7f361c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33559a45a4b0e08b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.180+0000 7f3627e00640 1 -- 172.21.15.132:0/3823680454 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c034070 con 0x7f36281516f0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.180+0000 7f3627e00640 1 -- 172.21.15.132:0/3823680454 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f361c030610 con 0x7f36281516f0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.180+0000 7f3627e00640 1 -- 172.21.15.132:0/3823680454 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c0309e0 con 0x7f36281516f0 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 -- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 msgr2=0x7f36280c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36280c7e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f361c0097a0 tx=0x7f361c02b960 comp rx=0 tx=0).stop 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 -- 172.21.15.132:0/3823680454 shutdown_connections 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36280c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 --2- 172.21.15.132:0/3823680454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36280c7e20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 -- 172.21.15.132:0/3823680454 >> 172.21.15.132:0/3823680454 conn(0x7f36280cc060 msgr2=0x7f36280cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:30.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 -- 172.21.15.132:0/3823680454 shutdown_connections 2024-03-27T06:21:30.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.181+0000 7f3630400640 1 -- 172.21.15.132:0/3823680454 wait complete. 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.183+0000 7f3630400640 1 Processor -- start 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.183+0000 7f3630400640 1 -- start start 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3628151ad0 0x7f36281e0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f362806bd80 con 0x7f3628151ad0 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3628072870 con 0x7f36281516f0 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f3630400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f36280c1080 con 0x7f3628157c80 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362da00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362e400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362e400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:39258/0 (socket says 172.21.15.132:39258) 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362e400640 1 -- 172.21.15.132:0/2895360004 learned_addr learned my addr 172.21.15.132:0/2895360004 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362da00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:55972/0 (socket says 172.21.15.132:55972) 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.184+0000 7f362d000640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3628151ad0 0x7f36281e0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 -- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 msgr2=0x7f36281e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 -- 172.21.15.132:0/2895360004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3628151ad0 msgr2=0x7f36281e0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3628151ad0 0x7f36281e0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 -- 172.21.15.132:0/2895360004 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36281ef4e0 con 0x7f3628157c80 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362da00640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f362e400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f362000efc0 tx=0x7f362000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620018020 con 0x7f3628157c80 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3620014480 con 0x7f3628157c80 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620009e60 con 0x7f3628157c80 2024-03-27T06:21:30.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36280cbc40 con 0x7f3628157c80 2024-03-27T06:21:30.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.185+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36281f1780 con 0x7f3628157c80 2024-03-27T06:21:30.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.187+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3628068840 con 0x7f3628157c80 2024-03-27T06:21:30.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.188+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f3620005d10 con 0x7f3628157c80 2024-03-27T06:21:30.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.189+0000 7f3625600640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 0x7f360806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.189+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f362008c500 con 0x7f3628157c80 2024-03-27T06:21:30.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.190+0000 7f362da00640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 0x7f360806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.190+0000 7f362da00640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 0x7f360806ea20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f361c036000 tx=0x7f361c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:30.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.194+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f3620022030 con 0x7f3628157c80 2024-03-27T06:21:30.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.563+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36281f1950 con 0x7f3628157c80 2024-03-27T06:21:30.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.565+0000 7f3625600640 1 -- 172.21.15.132:0/2895360004 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6056 (secure 0 0 0) 0x7f36281ef4e0 con 0x7f3628157c80 2024-03-27T06:21:30.555 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:30.555 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:29.204590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904]},"id":1}]} 2024-03-27T06:21:30.556 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 37 2024-03-27T06:21:30.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.569+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 msgr2=0x7f360806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.569+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 0x7f360806ea20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f361c036000 tx=0x7f361c03b040 comp rx=0 tx=0).stop 2024-03-27T06:21:30.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.569+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 msgr2=0x7f36281f1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.569+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f362000efc0 tx=0x7f362000c490 comp rx=0 tx=0).stop 2024-03-27T06:21:30.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 shutdown_connections 2024-03-27T06:21:30.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f3628157c80 0x7f36281f1ca0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3628151ad0 0x7f36281e0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f360806c560 0x7f360806ea20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 --2- 172.21.15.132:0/2895360004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36281516f0 0x7f36281e1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.570+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 >> 172.21.15.132:0/2895360004 conn(0x7f36280cc060 msgr2=0x7f362806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:30.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.571+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 shutdown_connections 2024-03-27T06:21:30.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.571+0000 7f3630400640 1 -- 172.21.15.132:0/2895360004 wait complete. 2024-03-27T06:21:30.595 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-03-27T06:21:30.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-03-27T06:21:30.595 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:30.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.976+0000 7fb038600640 1 Processor -- start 2024-03-27T06:21:30.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.977+0000 7fb038600640 1 -- start start 2024-03-27T06:21:30.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.979+0000 7fb038600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 0x7fb0301508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.979+0000 7fb038600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb030159e20 con 0x7fb030070ea0 2024-03-27T06:21:30.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.979+0000 7fb038600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0300c83c0 con 0x7fb03006dc50 2024-03-27T06:21:30.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.979+0000 7fb038600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb0300c8190 con 0x7fb0300c0ec0 2024-03-27T06:21:30.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.980+0000 7fb037000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 0x7fb0301508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.981+0000 7fb035200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb03006dc50 0x7fb03006d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56492/0 (socket says 172.21.15.132:56492) 2024-03-27T06:21:30.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.981+0000 7fb035200640 1 -- 172.21.15.132:0/3569573613 learned_addr learned my addr 172.21.15.132:0/3569573613 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:30.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.982+0000 7fb02fe00640 1 -- 172.21.15.132:0/3569573613 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 168115738 0 0) 0x7fb0300c83c0 con 0x7fb03006dc50 2024-03-27T06:21:30.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.982+0000 7fb02fe00640 1 -- 172.21.15.132:0/3569573613 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb018003520 con 0x7fb03006dc50 2024-03-27T06:21:30.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.983+0000 7fb037000640 1 -- 172.21.15.132:0/3569573613 >> v1:172.21.15.134:6789/0 conn(0x7fb03006dc50 legacy=0x7fb03006d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.983+0000 7fb037000640 1 -- 172.21.15.132:0/3569573613 >> v1:172.21.15.132:6789/0 conn(0x7fb030070ea0 legacy=0x7fb03006ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.983+0000 7fb037000640 1 -- 172.21.15.132:0/3569573613 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03015a090 con 0x7fb0300c0ec0 2024-03-27T06:21:30.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.986+0000 7fb037000640 1 --2- 172.21.15.132:0/3569573613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 0x7fb0301508c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb024009750 tx=0x7fb02402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9bd8ad98ef13f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:30.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.987+0000 7fb02fe00640 1 -- 172.21.15.132:0/3569573613 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024039070 con 0x7fb0300c0ec0 2024-03-27T06:21:30.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.988+0000 7fb02fe00640 1 -- 172.21.15.132:0/3569573613 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0240314f0 con 0x7fb0300c0ec0 2024-03-27T06:21:30.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.988+0000 7fb02fe00640 1 -- 172.21.15.132:0/3569573613 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c001670 con 0x7fb0300c0ec0 2024-03-27T06:21:30.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.989+0000 7fb038600640 1 -- 172.21.15.132:0/3569573613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 msgr2=0x7fb0301508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.989+0000 7fb038600640 1 --2- 172.21.15.132:0/3569573613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 0x7fb0301508c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb024009750 tx=0x7fb02402b790 comp rx=0 tx=0).stop 2024-03-27T06:21:30.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.990+0000 7fb038600640 1 -- 172.21.15.132:0/3569573613 shutdown_connections 2024-03-27T06:21:30.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.990+0000 7fb038600640 1 --2- 172.21.15.132:0/3569573613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb0300c0ec0 0x7fb0301508c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.990+0000 7fb038600640 1 -- 172.21.15.132:0/3569573613 >> 172.21.15.132:0/3569573613 conn(0x7fb0300c8910 msgr2=0x7fb0300c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:30.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.991+0000 7fb038600640 1 -- 172.21.15.132:0/3569573613 shutdown_connections 2024-03-27T06:21:30.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.991+0000 7fb038600640 1 -- 172.21.15.132:0/3569573613 wait complete. 2024-03-27T06:21:30.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.994+0000 7fb038600640 1 Processor -- start 2024-03-27T06:21:30.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.996+0000 7fb038600640 1 -- start start 2024-03-27T06:21:30.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb03006dc50 0x7fb03014b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 0x7fb03014c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:30.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb030157210 con 0x7fb0300c0ec0 2024-03-27T06:21:30.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb0300c8660 con 0x7fb03006dc50 2024-03-27T06:21:30.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.998+0000 7fb038600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb030157390 con 0x7fb030070ea0 2024-03-27T06:21:30.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.999+0000 7fb035200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.999+0000 7fb035200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:39282/0 (socket says 172.21.15.132:39282) 2024-03-27T06:21:30.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:30.999+0000 7fb035200640 1 -- 172.21.15.132:0/2944799965 learned_addr learned my addr 172.21.15.132:0/2944799965 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:30.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.000+0000 7fb037000640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 0x7fb03014c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:30.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.000+0000 7fb035200640 1 -- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb03006dc50 msgr2=0x7fb03014b6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T06:21:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.002+0000 7fb035200640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb03006dc50 0x7fb03014b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.002+0000 7fb035200640 1 -- 172.21.15.132:0/2944799965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 msgr2=0x7fb03014c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.002+0000 7fb035200640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 0x7fb03014c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.002+0000 7fb035200640 1 -- 172.21.15.132:0/2944799965 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0300caa40 con 0x7fb030070ea0 2024-03-27T06:21:30.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.002+0000 7fb037000640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 0x7fb03014c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:21:30.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.003+0000 7fb035200640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c8f0 tx=0x7fb01c00cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:30.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.004+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c018070 con 0x7fb030070ea0 2024-03-27T06:21:30.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.004+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0300cad20 con 0x7fb030070ea0 2024-03-27T06:21:30.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.005+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0300c52f0 con 0x7fb030070ea0 2024-03-27T06:21:30.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.010+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb01c0074e0 con 0x7fb030070ea0 2024-03-27T06:21:31.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.010+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c018070 con 0x7fb030070ea0 2024-03-27T06:21:31.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.018+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00c005290 con 0x7fb030070ea0 2024-03-27T06:21:31.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.020+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb01c01b070 con 0x7fb030070ea0 2024-03-27T06:21:31.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.024+0000 7fb02d600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 0x7fb01006ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:31.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.025+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fb01c08aab0 con 0x7fb030070ea0 2024-03-27T06:21:31.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.025+0000 7fb035c00640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 0x7fb01006ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:31.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.028+0000 7fb035c00640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 0x7fb01006ebf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb028007fd0 tx=0x7fb028007d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:31.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.033+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb01c058940 con 0x7fb030070ea0 2024-03-27T06:21:31.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.561+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb00c005850 con 0x7fb030070ea0 2024-03-27T06:21:31.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.566+0000 7fb02d600640 1 -- 172.21.15.132:0/2944799965 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6056 (secure 0 0 0) 0x7fb01c05cc40 con 0x7fb030070ea0 2024-03-27T06:21:31.557 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:31.561 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:29.204590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904]},"id":1}]} 2024-03-27T06:21:31.561 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 37 2024-03-27T06:21:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.575+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 msgr2=0x7fb01006ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.575+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 0x7fb01006ebf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb028007fd0 tx=0x7fb028007d80 comp rx=0 tx=0).stop 2024-03-27T06:21:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.575+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 msgr2=0x7fb03014ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:31.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.575+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c8f0 tx=0x7fb01c00cdb0 comp rx=0 tx=0).stop 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.576+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 shutdown_connections 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.576+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0300c0ec0 0x7fb03014c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.576+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb030070ea0 0x7fb03014ae60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.576+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb01006c730 0x7fb01006ebf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.576+0000 7fb038600640 1 --2- 172.21.15.132:0/2944799965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb03006dc50 0x7fb03014b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.577+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 >> 172.21.15.132:0/2944799965 conn(0x7fb0300c8910 msgr2=0x7fb0300c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:31.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.577+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 shutdown_connections 2024-03-27T06:21:31.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:31.577+0000 7fb038600640 1 -- 172.21.15.132:0/2944799965 wait complete. 2024-03-27T06:21:31.605 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_4904": { "addr": "172.21.15.134:6838/2682370774", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2682370774, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2682370774, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4904, "incarnation": 31, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_5284": { "addr": "172.21.15.134:6839/4090015809", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 4090015809, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 4090015809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5284, "incarnation": 37, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:21:29.204590+0000", "qdb_cluster": [ 4817, 4904 ], "qdb_leader": 4904, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4904, "mds_1": 4817, "mds_2": 5284 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:21:33.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:33.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.813+0000 7f8d9be00640 1 Processor -- start 2024-03-27T06:21:33.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.813+0000 7f8d9be00640 1 -- start start 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 0x7f8d941590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d9be00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9406b630 con 0x7f8d94073550 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d9be00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9406b8d0 con 0x7f8d94073970 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d9be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8d94152d30 con 0x7f8d94152950 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8d94073970 0x7f8d9406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:49238/0 (socket says 172.21.15.132:49238) 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d98a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8d94073550 0x7f8d94071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54574/0 (socket says 172.21.15.132:54574) 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99e00640 1 -- 172.21.15.132:0/1766860825 learned_addr learned my addr 172.21.15.132:0/1766860825 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99400640 1 --2- 172.21.15.132:0/1766860825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 0x7f8d941590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99400640 1 -- 172.21.15.132:0/1766860825 >> v1:172.21.15.134:6789/0 conn(0x7f8d94073970 legacy=0x7f8d9406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99400640 1 -- 172.21.15.132:0/1766860825 >> v1:172.21.15.132:6789/0 conn(0x7f8d94073550 legacy=0x7f8d94071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:33.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d99400640 1 -- 172.21.15.132:0/1766860825 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9406bbb0 con 0x7f8d94152950 2024-03-27T06:21:33.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.815+0000 7f8d93e00640 1 -- 172.21.15.132:0/1766860825 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910191803 0 0) 0x7f8d9406b630 con 0x7f8d94073550 2024-03-27T06:21:33.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d99400640 1 --2- 172.21.15.132:0/1766860825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 0x7f8d941590a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009750 tx=0x7f8d8802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1afb39c45b2e9da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:33.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d93e00640 1 -- 172.21.15.132:0/1766860825 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88037070 con 0x7f8d94152950 2024-03-27T06:21:33.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d93e00640 1 -- 172.21.15.132:0/1766860825 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d88030710 con 0x7f8d94152950 2024-03-27T06:21:33.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d93e00640 1 -- 172.21.15.132:0/1766860825 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88030ae0 con 0x7f8d94152950 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d9be00640 1 -- 172.21.15.132:0/1766860825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 msgr2=0x7f8d941590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.817+0000 7f8d9be00640 1 --2- 172.21.15.132:0/1766860825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 0x7f8d941590a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009750 tx=0x7f8d8802b730 comp rx=0 tx=0).stop 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.818+0000 7f8d9be00640 1 -- 172.21.15.132:0/1766860825 shutdown_connections 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.818+0000 7f8d9be00640 1 --2- 172.21.15.132:0/1766860825 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94152950 0x7f8d941590a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.818+0000 7f8d9be00640 1 -- 172.21.15.132:0/1766860825 >> 172.21.15.132:0/1766860825 conn(0x7f8d940cc060 msgr2=0x7f8d940cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.818+0000 7f8d9be00640 1 -- 172.21.15.132:0/1766860825 shutdown_connections 2024-03-27T06:21:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.818+0000 7f8d9be00640 1 -- 172.21.15.132:0/1766860825 wait complete. 2024-03-27T06:21:33.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.819+0000 7f8d9be00640 1 Processor -- start 2024-03-27T06:21:33.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 -- start start 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 0x7f8d941577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9406c450 con 0x7f8d94073550 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9406c170 con 0x7f8d94152950 2024-03-27T06:21:33.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.820+0000 7f8d9be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8d94071880 con 0x7f8d94073970 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 0x7f8d941577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d98a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50698/0 (socket says 172.21.15.132:50698) 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d98a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:49862/0 (socket says 172.21.15.132:49862) 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 0x7f8d941577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38384/0 (socket says 172.21.15.132:38384) 2024-03-27T06:21:33.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 -- 172.21.15.132:0/841500055 learned_addr learned my addr 172.21.15.132:0/841500055 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:33.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 -- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 msgr2=0x7f8d94144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:33.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:33.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 -- 172.21.15.132:0/841500055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 msgr2=0x7f8d941577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:33.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 0x7f8d941577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:33.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 -- 172.21.15.132:0/841500055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d941452e0 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d98a00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.821+0000 7f8d99e00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00caf0 tx=0x7f8d8c0020e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.822+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c018070 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.822+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00a760 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.822+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00ab50 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.822+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d94159750 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.822+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d941569d0 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.823+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d94159540 con 0x7f8d94152950 2024-03-27T06:21:33.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.824+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8d8c01b070 con 0x7f8d94152950 2024-03-27T06:21:33.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.825+0000 7f8d91600640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 0x7f8d7406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:33.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.825+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f8d8c08afc0 con 0x7f8d94152950 2024-03-27T06:21:33.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.825+0000 7f8d99400640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 0x7f8d7406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:33.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.826+0000 7f8d99400640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 0x7f8d7406e9f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d88034000 tx=0x7f8d88008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:33.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:33.833+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f8d94159540 con 0x7f8d94152950 2024-03-27T06:21:34.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.226+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d94068840 con 0x7f8d94152950 2024-03-27T06:21:34.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.227+0000 7f8d91600640 1 -- 172.21.15.132:0/841500055 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6056 (secure 0 0 0) 0x7f8d8c058db0 con 0x7f8d94152950 2024-03-27T06:21:34.217 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:34.217 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:29.204590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904]},"id":1}]} 2024-03-27T06:21:34.218 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 37 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 msgr2=0x7f8d7406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 0x7f8d7406e9f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d88034000 tx=0x7f8d88008030 comp rx=0 tx=0).stop 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 msgr2=0x7f8d941584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00caf0 tx=0x7f8d8c0020e0 comp rx=0 tx=0).stop 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 shutdown_connections 2024-03-27T06:21:34.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8d7406c530 0x7f8d7406e9f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d94152950 0x7f8d941584e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8d94073970 0x7f8d94144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 --2- 172.21.15.132:0/841500055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d94073550 0x7f8d941577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 >> 172.21.15.132:0/841500055 conn(0x7f8d940cc060 msgr2=0x7f8d940cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 shutdown_connections 2024-03-27T06:21:34.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:34.230+0000 7f8d9be00640 1 -- 172.21.15.132:0/841500055 wait complete. 2024-03-27T06:21:34.244 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:34.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:21:34.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:21:36.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:36.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.471+0000 7f8520600640 1 Processor -- start 2024-03-27T06:21:36.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.471+0000 7f8520600640 1 -- start start 2024-03-27T06:21:36.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.472+0000 7f8520600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8518154d80 0x7f851815b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.472+0000 7f8520600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f85180c72c0 0x7f851806b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.472+0000 7f8520600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85180c7e00 con 0x7f851806ba60 2024-03-27T06:21:36.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.472+0000 7f8520600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f85180c9140 con 0x7f85180c72c0 2024-03-27T06:21:36.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.472+0000 7f8520600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f8518067360 con 0x7f8518154d80 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8518154d80 0x7f851815b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851d200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f85180c72c0 0x7f851806b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8518154d80 0x7f851815b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50706/0 (socket says 172.21.15.132:50706) 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 -- 172.21.15.132:0/3074387606 learned_addr learned my addr 172.21.15.132:0/3074387606 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 -- 172.21.15.132:0/3074387606 >> v2:172.21.15.132:3300/0 conn(0x7f85180c72c0 msgr2=0x7f851806b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 --2- 172.21.15.132:0/3074387606 >> v2:172.21.15.132:3300/0 conn(0x7f85180c72c0 0x7f851806b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 -- 172.21.15.132:0/3074387606 >> v1:172.21.15.132:6789/0 conn(0x7f851806ba60 legacy=0x7f8518072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.464 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.473+0000 7f851dc00640 1 -- 172.21.15.132:0/3074387606 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85180c80a0 con 0x7f8518154d80 2024-03-27T06:21:36.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.474+0000 7f851dc00640 1 --2- 172.21.15.132:0/3074387606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f851815b3b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8518067ae0 tx=0x7f850c02b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2077ebf9c06b8739 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:36.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8517e00640 1 -- 172.21.15.132:0/3074387606 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c034070 con 0x7f8518154d80 2024-03-27T06:21:36.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8517e00640 1 -- 172.21.15.132:0/3074387606 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f850c030880 con 0x7f8518154d80 2024-03-27T06:21:36.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8517e00640 1 -- 172.21.15.132:0/3074387606 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c030c50 con 0x7f8518154d80 2024-03-27T06:21:36.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 -- 172.21.15.132:0/3074387606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 msgr2=0x7f851815b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 --2- 172.21.15.132:0/3074387606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f851815b3b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8518067ae0 tx=0x7f850c02b9c0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 -- 172.21.15.132:0/3074387606 shutdown_connections 2024-03-27T06:21:36.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 --2- 172.21.15.132:0/3074387606 >> v2:172.21.15.132:3300/0 conn(0x7f85180c72c0 0x7f851806b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 --2- 172.21.15.132:0/3074387606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f851815b3b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.475+0000 7f8520600640 1 -- 172.21.15.132:0/3074387606 >> 172.21.15.132:0/3074387606 conn(0x7f851806d7c0 msgr2=0x7f851806dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:36.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.476+0000 7f8520600640 1 -- 172.21.15.132:0/3074387606 shutdown_connections 2024-03-27T06:21:36.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.476+0000 7f8520600640 1 -- 172.21.15.132:0/3074387606 wait complete. 2024-03-27T06:21:36.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.477+0000 7f8520600640 1 Processor -- start 2024-03-27T06:21:36.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.477+0000 7f8520600640 1 -- start start 2024-03-27T06:21:36.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f8518159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85180c8660 con 0x7f851806ba60 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f851815b8f0 con 0x7f8518154d80 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f8520600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f85180c8940 con 0x7f85180c72c0 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38412/0 (socket says 172.21.15.132:38412) 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:49866/0 (socket says 172.21.15.132:49866) 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851dc00640 1 -- 172.21.15.132:0/3980279130 learned_addr learned my addr 172.21.15.132:0/3980279130 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.478+0000 7f851e600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f8518159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 -- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 msgr2=0x7f8518155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 -- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 msgr2=0x7f8518159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f8518159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 -- 172.21.15.132:0/3980279130 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85180c4f80 con 0x7f851806ba60 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851d200640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851e600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f8518159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:36.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.479+0000 7f851dc00640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f850c02ba00 tx=0x7f850c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.480+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c034040 con 0x7f851806ba60 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.480+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f850c039d90 con 0x7f851806ba60 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.480+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85180c6730 con 0x7f851806ba60 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.480+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c043610 con 0x7f851806ba60 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.480+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85180c5220 con 0x7f851806ba60 2024-03-27T06:21:36.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.481+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8518151240 con 0x7f851806ba60 2024-03-27T06:21:36.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.482+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f850c037070 con 0x7f851806ba60 2024-03-27T06:21:36.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.483+0000 7f8515600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 0x7f84f806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:36.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.483+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f850c0ada80 con 0x7f851806ba60 2024-03-27T06:21:36.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.483+0000 7f851d200640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 0x7f84f806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:36.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.484+0000 7f851d200640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 0x7f84f806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f85181f1500 tx=0x7f8504009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:36.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.489+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f8518151240 con 0x7f851806ba60 2024-03-27T06:21:36.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.788+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8518068840 con 0x7f851806ba60 2024-03-27T06:21:36.779 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.788+0000 7f8515600640 1 -- 172.21.15.132:0/3980279130 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6058 (secure 0 0 0) 0x7f850c0afe20 con 0x7f851806ba60 2024-03-27T06:21:36.779 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:36.779 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:34.215760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:resolve","state_seq":30,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817]},"id":1}]} 2024-03-27T06:21:36.780 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 38 2024-03-27T06:21:36.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 msgr2=0x7f84f806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 0x7f84f806e9f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f85181f1500 tx=0x7f8504009190 comp rx=0 tx=0).stop 2024-03-27T06:21:36.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 msgr2=0x7f85181554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f850c02ba00 tx=0x7f850c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 shutdown_connections 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f84f806c530 0x7f84f806e9f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8518154d80 0x7f8518159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f85180c72c0 0x7f8518155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 --2- 172.21.15.132:0/3980279130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f851806ba60 0x7f85181554c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.791+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 >> 172.21.15.132:0/3980279130 conn(0x7f851806d7c0 msgr2=0x7f851806f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.792+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 shutdown_connections 2024-03-27T06:21:36.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:36.792+0000 7f8520600640 1 -- 172.21.15.132:0/3980279130 wait complete. 2024-03-27T06:21:36.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:38.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:39.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.029+0000 7f5097600640 1 Processor -- start 2024-03-27T06:21:39.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.029+0000 7f5097600640 1 -- start start 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.030+0000 7f5097600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5090152950 0x7f50901590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.030+0000 7f5097600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 0x7f509006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.030+0000 7f5097600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509006b630 con 0x7f5090073550 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.030+0000 7f5097600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f5090152d30 con 0x7f5090152950 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.030+0000 7f5097600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f509006b8d0 con 0x7f5090073970 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5090152950 0x7f50901590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5090152950 0x7f50901590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38420/0 (socket says 172.21.15.132:38420) 2024-03-27T06:21:39.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 -- 172.21.15.132:0/1566709278 learned_addr learned my addr 172.21.15.132:0/1566709278 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:39.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5095600640 1 --2- 172.21.15.132:0/1566709278 >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 0x7f509006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 -- 172.21.15.132:0/1566709278 >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 msgr2=0x7f509006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:39.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 --2- 172.21.15.132:0/1566709278 >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 0x7f509006cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 -- 172.21.15.132:0/1566709278 >> v1:172.21.15.132:6789/0 conn(0x7f5090073550 legacy=0x7f5090071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5094c00640 1 -- 172.21.15.132:0/1566709278 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509006bbb0 con 0x7f5090152950 2024-03-27T06:21:39.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f508f400640 1 -- 172.21.15.132:0/1566709278 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456695782 0 0) 0x7f509006b630 con 0x7f5090073550 2024-03-27T06:21:39.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.031+0000 7f5095600640 1 --2- 172.21.15.132:0/1566709278 >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 0x7f509006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f5094c00640 1 --2- 172.21.15.132:0/1566709278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090152950 0x7f50901590a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5084009870 tx=0x7f508402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8071596321bbf7f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f508f400640 1 -- 172.21.15.132:0/1566709278 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084034070 con 0x7f5090152950 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f508f400640 1 -- 172.21.15.132:0/1566709278 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5084030850 con 0x7f5090152950 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f508f400640 1 -- 172.21.15.132:0/1566709278 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084030c20 con 0x7f5090152950 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f5097600640 1 -- 172.21.15.132:0/1566709278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090152950 msgr2=0x7f50901590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.033+0000 7f5097600640 1 --2- 172.21.15.132:0/1566709278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090152950 0x7f50901590a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5084009870 tx=0x7f508402b990 comp rx=0 tx=0).stop 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 -- 172.21.15.132:0/1566709278 shutdown_connections 2024-03-27T06:21:39.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 --2- 172.21.15.132:0/1566709278 >> v2:172.21.15.134:3300/0 conn(0x7f5090073970 0x7f509006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 --2- 172.21.15.132:0/1566709278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090152950 0x7f50901590a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 -- 172.21.15.132:0/1566709278 >> 172.21.15.132:0/1566709278 conn(0x7f50900cc060 msgr2=0x7f50900cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:39.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 -- 172.21.15.132:0/1566709278 shutdown_connections 2024-03-27T06:21:39.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.034+0000 7f5097600640 1 -- 172.21.15.132:0/1566709278 wait complete. 2024-03-27T06:21:39.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.036+0000 7f5097600640 1 Processor -- start 2024-03-27T06:21:39.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.037+0000 7f5097600640 1 -- start start 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090073970 0x7f509014b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5090152950 0x7f509006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f509006c450 con 0x7f5090073970 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f509006c170 con 0x7f5090073550 2024-03-27T06:21:39.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5097600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5090071880 con 0x7f5090152950 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5094c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5094c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50742/0 (socket says 172.21.15.132:50742) 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5094c00640 1 -- 172.21.15.132:0/3870924651 learned_addr learned my addr 172.21.15.132:0/3870924651 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f5095600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5090152950 0x7f509006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.038+0000 7f508fe00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090073970 0x7f509014b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 -- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5090152950 msgr2=0x7f509006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5090152950 0x7f509006cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 -- 172.21.15.132:0/3870924651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090073970 msgr2=0x7f509014b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090073970 0x7f509014b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 -- 172.21.15.132:0/3870924651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50900c57a0 con 0x7f5090073550 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f5094c00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.039+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084034040 con 0x7f5090073550 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.040+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5084039d90 con 0x7f5090073550 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.040+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084043610 con 0x7f5090073550 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.040+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50901ef0d0 con 0x7f5090073550 2024-03-27T06:21:39.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.040+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901ef690 con 0x7f5090073550 2024-03-27T06:21:39.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.041+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5090068840 con 0x7f5090073550 2024-03-27T06:21:39.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.042+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f508404a060 con 0x7f5090073550 2024-03-27T06:21:39.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.043+0000 7f508cc00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 0x7f507006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:39.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.043+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f50840ad710 con 0x7f5090073550 2024-03-27T06:21:39.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.043+0000 7f508fe00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 0x7f507006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:39.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.044+0000 7f508fe00640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 0x7f507006e9f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5090153730 tx=0x7f507c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:39.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.051+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f508403e070 con 0x7f5090073550 2024-03-27T06:21:39.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.300+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50900c30c0 con 0x7f5090073550 2024-03-27T06:21:39.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.300+0000 7f508cc00640 1 -- 172.21.15.132:0/3870924651 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6058 (secure 0 0 0) 0x7f508407bbf0 con 0x7f5090073550 2024-03-27T06:21:39.291 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:39.291 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:34.215760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:resolve","state_seq":30,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817]},"id":1}]} 2024-03-27T06:21:39.291 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 38 2024-03-27T06:21:39.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 msgr2=0x7f507006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 0x7f507006e9f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5090153730 tx=0x7f507c009040 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 msgr2=0x7f50900c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 shutdown_connections 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5090152950 0x7f509006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5090073970 0x7f509014b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f507006c530 0x7f507006e9f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 --2- 172.21.15.132:0/3870924651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090073550 0x7f50900c4f80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 >> 172.21.15.132:0/3870924651 conn(0x7f50900cc060 msgr2=0x7f50900c93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 shutdown_connections 2024-03-27T06:21:39.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:39.304+0000 7f5097600640 1 -- 172.21.15.132:0/3870924651 wait complete. 2024-03-27T06:21:39.314 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:39.714 INFO:tasks.workunit.client.0.smithi132.stderr:Updating files: 4% (585/12435) Updating files: 5% (622/12435) Updating files: 6% (747/12435) Updating files: 7% (871/12435) Updating files: 7% (963/12435) Updating files: 8% (995/12435) Updating files: 9% (1120/12435) Updating files: 10% (1244/12435) Updating files: 11% (1368/12435) Updating files: 11% (1480/12435) Updating files: 12% (1493/12435) Updating files: 13% (1617/12435) Updating files: 14% (1741/12435) Updating files: 14% (1825/12435) Updating files: 15% (1866/12435) Updating files: 16% (1990/12435) Updating files: 17% (2114/12435) Updating files: 18% (2239/12435) Updating files: 18% (2318/12435) Updating files: 19% (2363/12435) Updating files: 20% (2487/12435) Updating files: 21% (2612/12435) Updating files: 22% (2736/12435) Updating files: 22% (2762/12435) Updating files: 23% (2861/12435) Updating files: 24% (2985/12435) Updating files: 25% (3109/12435) Updating files: 25% (3213/12435) Updating files: 26% (3234/12435) Updating files: 27% (3358/12435) Updating files: 28% (3482/12435) Updating files: 29% (3607/12435) Updating files: 29% (3639/12435) Updating files: 30% (3731/12435) Updating files: 31% (3855/12435) Updating files: 32% (3980/12435) Updating files: 33% (4104/12435) Updating files: 33% (4161/12435) Updating files: 34% (4228/12435) Updating files: 35% (4353/12435) Updating files: 36% (4477/12435) Updating files: 37% (4601/12435) Updating files: 37% (4605/12435) Updating files: 38% (4726/12435) Updating files: 39% (4850/12435) Updating files: 40% (4974/12435) Updating files: 41% (5099/12435) Updating files: 41% (5150/12435) Updating files: 42% (5223/12435) Updating files: 43% (5348/12435) Updating files: 44% (5472/12435) Updating files: 44% (5512/12435) Updating files: 45% (5596/12435) Updating files: 46% (5721/12435) Updating files: 47% (5845/12435) Updating files: 47% (5855/12435) Updating files: 48% (5969/12435) Updating files: 49% (6094/12435) Updating files: 49% (6185/12435) Updating files: 50% (6218/12435) Updating files: 51% (6342/12435) Updating files: 52% (6467/12435) Updating files: 52% (6575/12435) Updating files: 53% (6591/12435) Updating files: 54% (6715/12435) Updating files: 55% (6840/12435) Updating files: 56% (6964/12435) Updating files: 57% (7088/12435) Updating files: 57% (7133/12435) Updating files: 58% (7213/12435) Updating files: 59% (7337/12435) Updating files: 60% (7461/12435) Updating files: 61% (7586/12435) Updating files: 61% (7600/12435) Updating files: 62% (7710/12435) Updating files: 62% (7782/12435) Updating files: 63% (7835/12435) Updating files: 64% (7959/12435) Updating files: 65% (8083/12435) Updating files: 65% (8157/12435) Updating files: 66% (8208/12435) Updating files: 67% (8332/12435) Updating files: 67% (8344/12435) Updating files: 68% (8456/12435) Updating files: 69% (8581/12435) Updating files: 70% (8705/12435) Updating files: 70% (8746/12435) Updating files: 71% (8829/12435) Updating files: 72% (8954/12435) Updating files: 72% (8988/12435) Updating files: 73% (9078/12435) Updating files: 74% (9202/12435) Updating files: 75% (9327/12435) Updating files: 75% (9440/12435) Updating files: 76% (9451/12435) Updating files: 77% (9575/12435) Updating files: 78% (9700/12435) Updating files: 79% (9824/12435) Updating files: 79% (9898/12435) Updating files: 79% (9928/12435) Updating files: 80% (9948/12435) Updating files: 81% (10073/12435) Updating files: 82% (10197/12435) Updating files: 82% (10294/12435) Updating files: 83% (10322/12435) Updating files: 84% (10446/12435) Updating files: 85% (10570/12435) Updating files: 86% (10695/12435) Updating files: 86% (10708/12435) Updating files: 87% (10819/12435) Updating files: 88% (10943/12435) Updating files: 89% (11068/12435) Updating files: 90% (11192/12435) Updating files: 90% (11215/12435) Updating files: 91% (11316/12435) Updating files: 92% (11441/12435) Updating files: 93% (11565/12435) Updating files: 94% (11689/12435) Updating files: 94% (11765/12435) Updating files: 95% (11814/12435) Updating files: 96% (11938/12435) Updating files: 97% (12062/12435) Updating files: 97% (12109/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-27T06:21:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-27T06:21:39.801 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-03-27T06:21:39.801 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-27T06:21:40.938 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi132 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-27T06:21:41.194 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b631062c33ed425682ae5636c619af39 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_155268c4e432a12433aa833f174f9fe3b1016ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi132 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-27T06:21:41.197 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-03-27T06:21:41.207 INFO:tasks.mds_thrash:joining mds_thrasher 2024-03-27T06:21:41.315 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:41.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.530+0000 7f5f55c00640 1 Processor -- start 2024-03-27T06:21:41.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.530+0000 7f5f55c00640 1 -- start start 2024-03-27T06:21:41.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5f5015b120 0x7f5f500ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5f500c9c80 0x7f5f500ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f5f5006ed40 con 0x7f5f500c9c80 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5f50158670 con 0x7f5f5015b120 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.531+0000 7f5f55c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5006e2f0 con 0x7f5f50158290 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4e000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5f5015b120 0x7f5f500ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4f400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5f500c9c80 0x7f5f500ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:49876/0 (socket says 172.21.15.132:49876) 2024-03-27T06:21:41.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 -- 172.21.15.132:0/614718843 learned_addr learned my addr 172.21.15.132:0/614718843 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 -- 172.21.15.132:0/614718843 >> v2:172.21.15.134:3300/0 conn(0x7f5f5015b120 msgr2=0x7f5f500ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/614718843 >> v2:172.21.15.134:3300/0 conn(0x7f5f5015b120 0x7f5f500ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 -- 172.21.15.132:0/614718843 >> v2:172.21.15.132:3300/0 conn(0x7f5f500c9c80 msgr2=0x7f5f500ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/614718843 >> v2:172.21.15.132:3300/0 conn(0x7f5f500c9c80 0x7f5f500ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.532+0000 7f5f4ea00640 1 -- 172.21.15.132:0/614718843 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5006c6d0 con 0x7f5f50158290 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.533+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/614718843 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5f50072af0 tx=0x7f5f4002b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937707d2096b1b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f4d600640 1 -- 172.21.15.132:0/614718843 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40035020 con 0x7f5f50158290 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f4d600640 1 -- 172.21.15.132:0/614718843 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4002d040 con 0x7f5f50158290 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f4d600640 1 -- 172.21.15.132:0/614718843 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40031a80 con 0x7f5f50158290 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 -- 172.21.15.132:0/614718843 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 msgr2=0x7f5f5015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 --2- 172.21.15.132:0/614718843 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5f50072af0 tx=0x7f5f4002b900 comp rx=0 tx=0).stop 2024-03-27T06:21:41.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 -- 172.21.15.132:0/614718843 shutdown_connections 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 --2- 172.21.15.132:0/614718843 >> v2:172.21.15.132:3300/0 conn(0x7f5f500c9c80 0x7f5f500ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 --2- 172.21.15.132:0/614718843 >> v2:172.21.15.134:3300/0 conn(0x7f5f5015b120 0x7f5f500ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 --2- 172.21.15.132:0/614718843 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f5015abe0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 -- 172.21.15.132:0/614718843 >> 172.21.15.132:0/614718843 conn(0x7f5f50067280 msgr2=0x7f5f50067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 -- 172.21.15.132:0/614718843 shutdown_connections 2024-03-27T06:21:41.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.534+0000 7f5f55c00640 1 -- 172.21.15.132:0/614718843 wait complete. 2024-03-27T06:21:41.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.536+0000 7f5f55c00640 1 Processor -- start 2024-03-27T06:21:41.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.536+0000 7f5f55c00640 1 -- start start 2024-03-27T06:21:41.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 0x7f5f50147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5006e5d0 con 0x7f5f501f0a60 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5006f690 con 0x7f5f500c9c80 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f55c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5006f1d0 con 0x7f5f50158290 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4e000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 0x7f5f50147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4e000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:49888/0 (socket says 172.21.15.132:49888) 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4e000640 1 -- 172.21.15.132:0/4012186280 learned_addr learned my addr 172.21.15.132:0/4012186280 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.537+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:50764/0 (socket says 172.21.15.132:50764) 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 -- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 msgr2=0x7f5f500ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 -- 172.21.15.132:0/4012186280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 msgr2=0x7f5f50147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:41.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 0x7f5f50147680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 -- 172.21.15.132:0/4012186280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f500b3220 con 0x7f5f50158290 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4f400640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 0x7f5f50147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f4e000640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00e9b0 tx=0x7f5f3c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c017070 con 0x7f5f50158290 2024-03-27T06:21:41.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f3c00c9a0 con 0x7f5f50158290 2024-03-27T06:21:41.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f500b2f80 con 0x7f5f50158290 2024-03-27T06:21:41.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c00cd90 con 0x7f5f50158290 2024-03-27T06:21:41.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.538+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f501f3bc0 con 0x7f5f50158290 2024-03-27T06:21:41.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.540+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f50068840 con 0x7f5f50158290 2024-03-27T06:21:41.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.540+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5f3c01a070 con 0x7f5f50158290 2024-03-27T06:21:41.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.541+0000 7f5f46a00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 0x7f5f3006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:41.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.541+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f5f3c08b8b0 con 0x7f5f50158290 2024-03-27T06:21:41.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.541+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 0x7f5f3006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:41.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.542+0000 7f5f4ea00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 0x7f5f3006e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f40036000 tx=0x7f5f4003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:41.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.549+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5f3c059610 con 0x7f5f50158290 2024-03-27T06:21:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.786+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f5006f8f0 con 0x7f5f50158290 2024-03-27T06:21:41.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.787+0000 7f5f46a00640 1 -- 172.21.15.132:0/4012186280 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6060 (secure 0 0 0) 0x7f5f3c05d910 con 0x7f5f50158290 2024-03-27T06:21:41.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:41.777 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:39.870688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904]},"id":1}]} 2024-03-27T06:21:41.777 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 39 2024-03-27T06:21:41.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 msgr2=0x7f5f3006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:41.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 0x7f5f3006e9f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5f40036000 tx=0x7f5f4003c040 comp rx=0 tx=0).stop 2024-03-27T06:21:41.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 msgr2=0x7f5f501f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:41.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00e9b0 tx=0x7f5f3c00ee70 comp rx=0 tx=0).stop 2024-03-27T06:21:41.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 shutdown_connections 2024-03-27T06:21:41.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f501f0a60 0x7f5f50147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5f50158290 0x7f5f501f0520 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5f3006c530 0x7f5f3006e9f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 --2- 172.21.15.132:0/4012186280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f500c9c80 0x7f5f500ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 >> 172.21.15.132:0/4012186280 conn(0x7f5f50067280 msgr2=0x7f5f500c89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.789+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 shutdown_connections 2024-03-27T06:21:41.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:41.790+0000 7f5f55c00640 1 -- 172.21.15.132:0/4012186280 wait complete. 2024-03-27T06:21:41.797 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:43.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:44.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.013+0000 7f7f02000640 1 Processor -- start 2024-03-27T06:21:44.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.013+0000 7f7f02000640 1 -- start start 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7f02000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 0x7f7efc06fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7f02000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7efc0b0d10 0x7f7efc0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7f02000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7efc06f6c0 con 0x7f7efc152c60 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7f02000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f7efc06f250 con 0x7f7efc0b0d10 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7f02000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7efc0c3760 con 0x7f7efc159c30 2024-03-27T06:21:44.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 0x7f7efc06fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7efb400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7efc152c60 0x7f7efc0c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53388/0 (socket says 172.21.15.132:53388) 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7efb400640 1 -- 172.21.15.132:0/1461583206 learned_addr learned my addr 172.21.15.132:0/1461583206 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7efbe00640 1 --2- 172.21.15.132:0/1461583206 >> v2:172.21.15.132:3300/0 conn(0x7f7efc0b0d10 0x7f7efc0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1461583206 >> v2:172.21.15.132:3300/0 conn(0x7f7efc0b0d10 msgr2=0x7f7efc0ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7ef3e00640 1 --2- 172.21.15.132:0/1461583206 >> v2:172.21.15.132:3300/0 conn(0x7f7efc0b0d10 0x7f7efc0ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1461583206 >> v1:172.21.15.134:6789/0 conn(0x7f7efc152c60 legacy=0x7f7efc0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7efaa00640 1 -- 172.21.15.132:0/1461583206 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445156120 0 0) 0x7f7efc06f6c0 con 0x7f7efc152c60 2024-03-27T06:21:44.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.015+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1461583206 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc15c7e0 con 0x7f7efc159c30 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.016+0000 7f7ef3e00640 1 --2- 172.21.15.132:0/1461583206 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 0x7f7efc06fb10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097a0 tx=0x7f7ee402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7526ffcad259f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7efaa00640 1 -- 172.21.15.132:0/1461583206 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4035020 con 0x7f7efc159c30 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7efaa00640 1 -- 172.21.15.132:0/1461583206 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee4031830 con 0x7f7efc159c30 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7efaa00640 1 -- 172.21.15.132:0/1461583206 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4031c00 con 0x7f7efc159c30 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 -- 172.21.15.132:0/1461583206 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 msgr2=0x7f7efc06fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 --2- 172.21.15.132:0/1461583206 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 0x7f7efc06fb10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097a0 tx=0x7f7ee402b8d0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 -- 172.21.15.132:0/1461583206 shutdown_connections 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 --2- 172.21.15.132:0/1461583206 >> v2:172.21.15.132:3300/0 conn(0x7f7efc0b0d10 0x7f7efc0ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 --2- 172.21.15.132:0/1461583206 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc159c30 0x7f7efc06fb10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 -- 172.21.15.132:0/1461583206 >> 172.21.15.132:0/1461583206 conn(0x7f7efc067280 msgr2=0x7f7efc067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 -- 172.21.15.132:0/1461583206 shutdown_connections 2024-03-27T06:21:44.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.017+0000 7f7f02000640 1 -- 172.21.15.132:0/1461583206 wait complete. 2024-03-27T06:21:44.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.019+0000 7f7f02000640 1 Processor -- start 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.019+0000 7f7f02000640 1 -- start start 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 0x7f7efc0b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7efc159c30 0x7f7efc1e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc0c3320 con 0x7f7efc159c30 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc06eee0 con 0x7f7efc152c60 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7f02000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7efc0c34a0 con 0x7f7efc0b0d10 2024-03-27T06:21:44.012 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7efb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 0x7f7efc0b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36614/0 (socket says 172.21.15.132:36614) 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1087599563 learned_addr learned my addr 172.21.15.132:0/1087599563 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.020+0000 7f7efbe00640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7efc159c30 0x7f7efc1e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 msgr2=0x7f7efc0b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 0x7f7efc0b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.013 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1087599563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7efc159c30 msgr2=0x7f7efc1e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7efc159c30 0x7f7efc1e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 -- 172.21.15.132:0/1087599563 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc0caf80 con 0x7f7efc152c60 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7efb400640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 0x7f7efc0b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3e00640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7ee402bee0 tx=0x7f7ee40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4035040 con 0x7f7efc152c60 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee403b920 con 0x7f7efc152c60 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7efc1e3bc0 con 0x7f7efc152c60 2024-03-27T06:21:44.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.021+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee403bd10 con 0x7f7efc152c60 2024-03-27T06:21:44.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.022+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7efc1e4180 con 0x7f7efc152c60 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.023+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7efc068840 con 0x7f7efc152c60 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.023+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7ee4038020 con 0x7f7efc152c60 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.025+0000 7f7ef3400640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 0x7f7ed806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.025+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f7ee40ad930 con 0x7f7efc152c60 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.025+0000 7f7efb400640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 0x7f7ed806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.026+0000 7f7efb400640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 0x7f7ed806ea20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7eec007fd0 tx=0x7f7eec007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:44.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.034+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f7ee4031800 con 0x7f7efc152c60 2024-03-27T06:21:44.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.345+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7efc06f8f0 con 0x7f7efc152c60 2024-03-27T06:21:44.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.345+0000 7f7ef3400640 1 -- 172.21.15.132:0/1087599563 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6060 (secure 0 0 0) 0x7f7ee407b6a0 con 0x7f7efc152c60 2024-03-27T06:21:44.336 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:44.336 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:39.870688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904]},"id":1}]} 2024-03-27T06:21:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 39 2024-03-27T06:21:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 msgr2=0x7f7ed806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 0x7f7ed806ea20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7eec007fd0 tx=0x7f7eec007de0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 msgr2=0x7f7efc0cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:44.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7ee402bee0 tx=0x7f7ee40046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 shutdown_connections 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7efc159c30 0x7f7efc1e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7ed806c560 0x7f7ed806ea20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7efc152c60 0x7f7efc0cda90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 --2- 172.21.15.132:0/1087599563 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7efc0b0d10 0x7f7efc0b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 >> 172.21.15.132:0/1087599563 conn(0x7f7efc067280 msgr2=0x7f7efc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 shutdown_connections 2024-03-27T06:21:44.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:44.349+0000 7f7f02000640 1 -- 172.21.15.132:0/1087599563 wait complete. 2024-03-27T06:21:44.367 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-27T06:14:02.985226+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4817": { "addr": "172.21.15.132:6836/2917670201", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6834", "nonce": 2917670201, "type": "v2" }, { "addr": "172.21.15.132:6836", "nonce": 2917670201, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4817, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_4904": { "addr": "172.21.15.134:6838/2682370774", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6836", "nonce": 2682370774, "type": "v2" }, { "addr": "172.21.15.134:6838", "nonce": 2682370774, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4904, "incarnation": 31, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_5284": { "addr": "172.21.15.134:6839/4090015809", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6837", "nonce": 4090015809, "type": "v2" }, { "addr": "172.21.15.134:6839", "nonce": 4090015809, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5284, "incarnation": 37, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-27T06:21:39.870688+0000", "qdb_cluster": [ 4817, 4904 ], "qdb_leader": 4904, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4904, "mds_1": 4817, "mds_2": 5284 } } } ], "standbys": [ { "addr": "172.21.15.134:6835/4272768803", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6834", "nonce": 4272768803, "type": "v2" }, { "addr": "172.21.15.134:6835", "nonce": 4272768803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4226, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.132:6833/3378771917", "addrs": { "addrvec": [ { "addr": "172.21.15.132:6832", "nonce": 3378771917, "type": "v2" }, { "addr": "172.21.15.132:6833", "nonce": 3378771917, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4238, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-27T06:21:46.368 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:46.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.625+0000 7fce6cc00640 1 Processor -- start 2024-03-27T06:21:46.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.625+0000 7fce6cc00640 1 -- start start 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce6cc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fce68152930 0x7fce680c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce6cc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce680c4cd0 con 0x7fce6806f960 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce6cc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce68158cc0 con 0x7fce680c8af0 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce6cc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fce680c3800 con 0x7fce68152930 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fce68152930 0x7fce680c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fce680c8af0 0x7fce681585b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53390/0 (socket says 172.21.15.132:53390) 2024-03-27T06:21:46.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65400640 1 -- 172.21.15.132:0/3422101913 learned_addr learned my addr 172.21.15.132:0/3422101913 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:46.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce64a00640 1 -- 172.21.15.132:0/3422101913 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613512733 0 0) 0x7fce68158cc0 con 0x7fce680c8af0 2024-03-27T06:21:46.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce64a00640 1 -- 172.21.15.132:0/3422101913 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce4c003520 con 0x7fce680c8af0 2024-03-27T06:21:46.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65e00640 1 -- 172.21.15.132:0/3422101913 >> v1:172.21.15.134:6789/0 conn(0x7fce680c8af0 legacy=0x7fce681585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65e00640 1 -- 172.21.15.132:0/3422101913 >> v1:172.21.15.132:6789/0 conn(0x7fce6806f960 legacy=0x7fce680ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.627+0000 7fce65e00640 1 -- 172.21.15.132:0/3422101913 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6806eea0 con 0x7fce68152930 2024-03-27T06:21:46.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce65e00640 1 --2- 172.21.15.132:0/3422101913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce68152930 0x7fce680c8230 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce580097a0 tx=0x7fce5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85e621e3f7dfcccc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:46.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce64a00640 1 -- 172.21.15.132:0/3422101913 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58035020 con 0x7fce68152930 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce64a00640 1 -- 172.21.15.132:0/3422101913 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce58031830 con 0x7fce68152930 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce64a00640 1 -- 172.21.15.132:0/3422101913 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58031c00 con 0x7fce68152930 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce6cc00640 1 -- 172.21.15.132:0/3422101913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce68152930 msgr2=0x7fce680c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3422101913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce68152930 0x7fce680c8230 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce580097a0 tx=0x7fce5802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce6cc00640 1 -- 172.21.15.132:0/3422101913 shutdown_connections 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3422101913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce68152930 0x7fce680c8230 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.629+0000 7fce6cc00640 1 -- 172.21.15.132:0/3422101913 >> 172.21.15.132:0/3422101913 conn(0x7fce68070dd0 msgr2=0x7fce680711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.630+0000 7fce6cc00640 1 -- 172.21.15.132:0/3422101913 shutdown_connections 2024-03-27T06:21:46.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.630+0000 7fce6cc00640 1 -- 172.21.15.132:0/3422101913 wait complete. 2024-03-27T06:21:46.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.631+0000 7fce6cc00640 1 Processor -- start 2024-03-27T06:21:46.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.631+0000 7fce6cc00640 1 -- start start 2024-03-27T06:21:46.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:46.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 0x7fce680cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fce68152930 0x7fce681f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce6806ec50 con 0x7fce680c8af0 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fce68152d10 con 0x7fce6806f960 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.632+0000 7fce6cc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fce680c3520 con 0x7fce68152930 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:36628/0 (socket says 172.21.15.132:36628) 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 -- 172.21.15.132:0/3246484163 learned_addr learned my addr 172.21.15.132:0/3246484163 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce66800640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fce68152930 0x7fce681f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:46.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65400640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 0x7fce680cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 -- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fce68152930 msgr2=0x7fce681f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fce68152930 0x7fce681f00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 -- 172.21.15.132:0/3246484163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 msgr2=0x7fce680cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 0x7fce680cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 -- 172.21.15.132:0/3246484163 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce681f0be0 con 0x7fce6806f960 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65400640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 0x7fce680cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.633+0000 7fce65e00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce58034040 tx=0x7fce580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:46.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.634+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58040070 con 0x7fce6806f960 2024-03-27T06:21:46.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.634+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5803a910 con 0x7fce6806f960 2024-03-27T06:21:46.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.634+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58035040 con 0x7fce6806f960 2024-03-27T06:21:46.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.634+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce681f0ec0 con 0x7fce6806f960 2024-03-27T06:21:46.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.634+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681568c0 con 0x7fce6806f960 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.635+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce68068840 con 0x7fce6806f960 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.636+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fce58038020 con 0x7fce6806f960 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.637+0000 7fce5e000640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 0x7fce4806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.637+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fce580ad210 con 0x7fce6806f960 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.637+0000 7fce65400640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 0x7fce4806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:46.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.637+0000 7fce65400640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 0x7fce4806e9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce68159930 tx=0x7fce54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:46.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.641+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fce58031800 con 0x7fce6806f960 2024-03-27T06:21:46.871 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.882+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce68069020 con 0x7fce6806f960 2024-03-27T06:21:46.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.883+0000 7fce5e000640 1 -- 172.21.15.132:0/3246484163 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6057 (secure 0 0 0) 0x7fce5807b530 con 0x7fce6806f960 2024-03-27T06:21:46.873 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:46.873 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:44.888215+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817]},"id":1}]} 2024-03-27T06:21:46.873 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 40 2024-03-27T06:21:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.885+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 msgr2=0x7fce4806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.885+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 0x7fce4806e9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce68159930 tx=0x7fce54009040 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.885+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 msgr2=0x7fce6814ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.885+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce58034040 tx=0x7fce580046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 shutdown_connections 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fce68152930 0x7fce681f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce680c8af0 0x7fce680cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fce4806c530 0x7fce4806e9f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 --2- 172.21.15.132:0/3246484163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce6806f960 0x7fce6814ab70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 >> 172.21.15.132:0/3246484163 conn(0x7fce68070dd0 msgr2=0x7fce68151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:46.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 shutdown_connections 2024-03-27T06:21:46.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:46.886+0000 7fce6cc00640 1 -- 172.21.15.132:0/3246484163 wait complete. 2024-03-27T06:21:46.895 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:48.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:49.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.108+0000 7fd5faa00640 1 Processor -- start 2024-03-27T06:21:49.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.108+0000 7fd5faa00640 1 -- start start 2024-03-27T06:21:49.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5faa00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 0x7fd5f406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5faa00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f415de10 con 0x7fd5f41549a0 2024-03-27T06:21:49.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5faa00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f40cada0 con 0x7fd5f40cd6a0 2024-03-27T06:21:49.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5faa00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f40cdad0 con 0x7fd5f406be10 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5f8c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 0x7fd5f406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5fa000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5f9600640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd5f41549a0 0x7fd5f40cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:34196/0 (socket says 172.21.15.132:34196) 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5f9600640 1 -- 172.21.15.132:0/1625086932 learned_addr learned my addr 172.21.15.132:0/1625086932 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.110+0000 7fd5fa000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:52140/0 (socket says 172.21.15.132:52140) 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.111+0000 7fd5fa000640 1 -- 172.21.15.132:0/1625086932 >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 msgr2=0x7fd5f406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:49.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.111+0000 7fd5fa000640 1 --2- 172.21.15.132:0/1625086932 >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 0x7fd5f406b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.111+0000 7fd5fa000640 1 -- 172.21.15.132:0/1625086932 >> v1:172.21.15.132:6789/0 conn(0x7fd5f41549a0 legacy=0x7fd5f40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.111+0000 7fd5fa000640 1 -- 172.21.15.132:0/1625086932 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f415e0b0 con 0x7fd5f406be10 2024-03-27T06:21:49.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.111+0000 7fd5f8c00640 1 --2- 172.21.15.132:0/1625086932 >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 0x7fd5f406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:49.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5fa000640 1 --2- 172.21.15.132:0/1625086932 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec02bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aca00417c86eabf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:49.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5f3e00640 1 -- 172.21.15.132:0/1625086932 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec039070 con 0x7fd5f406be10 2024-03-27T06:21:49.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5f3e00640 1 -- 172.21.15.132:0/1625086932 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ec02ccf0 con 0x7fd5f406be10 2024-03-27T06:21:49.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5f3e00640 1 -- 172.21.15.132:0/1625086932 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec034660 con 0x7fd5f406be10 2024-03-27T06:21:49.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5faa00640 1 -- 172.21.15.132:0/1625086932 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 msgr2=0x7fd5f415adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.112+0000 7fd5faa00640 1 --2- 172.21.15.132:0/1625086932 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec02bf40 comp rx=0 tx=0).stop 2024-03-27T06:21:49.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 -- 172.21.15.132:0/1625086932 shutdown_connections 2024-03-27T06:21:49.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 --2- 172.21.15.132:0/1625086932 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f406be10 0x7fd5f415adb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 --2- 172.21.15.132:0/1625086932 >> v2:172.21.15.132:3300/0 conn(0x7fd5f40cd6a0 0x7fd5f406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 -- 172.21.15.132:0/1625086932 >> 172.21.15.132:0/1625086932 conn(0x7fd5f406d8d0 msgr2=0x7fd5f406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:49.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 -- 172.21.15.132:0/1625086932 shutdown_connections 2024-03-27T06:21:49.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.113+0000 7fd5faa00640 1 -- 172.21.15.132:0/1625086932 wait complete. 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.114+0000 7fd5faa00640 1 Processor -- start 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.114+0000 7fd5faa00640 1 -- start start 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 0x7fd5f415c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 0x7fd5f41f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f415e950 con 0x7fd5f406be10 2024-03-27T06:21:49.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f415e670 con 0x7fd5f40cd6a0 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.115+0000 7fd5faa00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f415ec30 con 0x7fd5f41549a0 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 0x7fd5f415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5fa000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f8c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 0x7fd5f41f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5fa000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:52144/0 (socket says 172.21.15.132:52144) 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5fa000640 1 -- 172.21.15.132:0/3349131320 learned_addr learned my addr 172.21.15.132:0/3349131320 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 -- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 msgr2=0x7fd5f40b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 -- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 msgr2=0x7fd5f41f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 0x7fd5f41f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 -- 172.21.15.132:0/3349131320 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f415c9d0 con 0x7fd5f406be10 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5fa000640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f8c00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 0x7fd5f41f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:49.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.116+0000 7fd5f9600640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 0x7fd5f415c490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000f60 tx=0x7fd5e800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:49.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.117+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8017070 con 0x7fd5f406be10 2024-03-27T06:21:49.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.117+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e8005ce0 con 0x7fd5f406be10 2024-03-27T06:21:49.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.117+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8007310 con 0x7fd5f406be10 2024-03-27T06:21:49.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.117+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f41f0bc0 con 0x7fd5f406be10 2024-03-27T06:21:49.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.117+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f41f1180 con 0x7fd5f406be10 2024-03-27T06:21:49.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.118+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f40686c0 con 0x7fd5f406be10 2024-03-27T06:21:49.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.119+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fd5e801a070 con 0x7fd5f406be10 2024-03-27T06:21:49.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.120+0000 7fd5f1600640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 0x7fd5d406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:49.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.121+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fd5e808aa80 con 0x7fd5f406be10 2024-03-27T06:21:49.111 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.121+0000 7fd5f8c00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 0x7fd5d406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:49.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.121+0000 7fd5f8c00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 0x7fd5d406e9f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009560 tx=0x7fd5e0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:49.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.125+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fd5e8058880 con 0x7fd5f406be10 2024-03-27T06:21:49.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.366+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5f4075ab0 con 0x7fd5f406be10 2024-03-27T06:21:49.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.367+0000 7fd5f1600640 1 -- 172.21.15.132:0/3349131320 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6057 (secure 0 0 0) 0x7fd5e805cb80 con 0x7fd5f406be10 2024-03-27T06:21:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:49.357 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:44.888215+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4904,4817]},"id":1}]} 2024-03-27T06:21:49.357 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 40 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 msgr2=0x7fd5d406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 0x7fd5d406e9f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009560 tx=0x7fd5e0009190 comp rx=0 tx=0).stop 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 msgr2=0x7fd5f415c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 0x7fd5f415c490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000f60 tx=0x7fd5e800c760 comp rx=0 tx=0).stop 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 shutdown_connections 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fd5f41549a0 0x7fd5f40b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fd5d406c530 0x7fd5d406e9f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5f40cd6a0 0x7fd5f41f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 --2- 172.21.15.132:0/3349131320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5f406be10 0x7fd5f415c490 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:49.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 >> 172.21.15.132:0/3349131320 conn(0x7fd5f406d8d0 msgr2=0x7fd5f40cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:49.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 shutdown_connections 2024-03-27T06:21:49.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:49.370+0000 7fd5faa00640 1 -- 172.21.15.132:0/3349131320 wait complete. 2024-03-27T06:21:49.378 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-03-27T06:21:51.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:51.535 INFO:tasks.ceph.mds.b.smithi134.stderr:2024-03-27T06:21:51.546+0000 7fa4708d3640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-27T06:21:51.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.613+0000 7f9548000640 1 Processor -- start 2024-03-27T06:21:51.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.613+0000 7f9548000640 1 -- start start 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9548000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f9540162e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9548000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954006b520 con 0x7f9540151ad0 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9548000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954006b7c0 con 0x7f95401516f0 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9548000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9540072be0 con 0x7f9540157c80 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9545600640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f95401516f0 0x7f95400c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:53396/0 (socket says 172.21.15.132:53396) 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9545600640 1 -- 172.21.15.132:0/4249659169 learned_addr learned my addr 172.21.15.132:0/4249659169 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:51.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.614+0000 7f9546000640 1 --2- 172.21.15.132:0/4249659169 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f9540162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.615+0000 7f953fe00640 1 -- 172.21.15.132:0/4249659169 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054787521 0 0) 0x7f954006b7c0 con 0x7f95401516f0 2024-03-27T06:21:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.615+0000 7f9546000640 1 -- 172.21.15.132:0/4249659169 >> v1:172.21.15.134:6789/0 conn(0x7f95401516f0 legacy=0x7f95400c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.615+0000 7f9546000640 1 -- 172.21.15.132:0/4249659169 >> v1:172.21.15.132:6789/0 conn(0x7f9540151ad0 legacy=0x7f9540157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:51.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.615+0000 7f9546000640 1 -- 172.21.15.132:0/4249659169 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954006baa0 con 0x7f9540157c80 2024-03-27T06:21:51.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.616+0000 7f9546000640 1 --2- 172.21.15.132:0/4249659169 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f9540162e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9540072d60 tx=0x7f953802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56c842d69018751e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:51.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.616+0000 7f953fe00640 1 -- 172.21.15.132:0/4249659169 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538039070 con 0x7f9540157c80 2024-03-27T06:21:51.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.616+0000 7f953fe00640 1 -- 172.21.15.132:0/4249659169 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f953802cb10 con 0x7f9540157c80 2024-03-27T06:21:51.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.616+0000 7f953fe00640 1 -- 172.21.15.132:0/4249659169 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538034440 con 0x7f9540157c80 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 -- 172.21.15.132:0/4249659169 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 msgr2=0x7f9540162e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 --2- 172.21.15.132:0/4249659169 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f9540162e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9540072d60 tx=0x7f953802bde0 comp rx=0 tx=0).stop 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 -- 172.21.15.132:0/4249659169 shutdown_connections 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 --2- 172.21.15.132:0/4249659169 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f9540162e20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 -- 172.21.15.132:0/4249659169 >> 172.21.15.132:0/4249659169 conn(0x7f95400cc060 msgr2=0x7f95400cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 -- 172.21.15.132:0/4249659169 shutdown_connections 2024-03-27T06:21:51.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.617+0000 7f9548000640 1 -- 172.21.15.132:0/4249659169 wait complete. 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.619+0000 7f9548000640 1 Processor -- start 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.619+0000 7f9548000640 1 -- start start 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 0x7f954014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9540151ad0 0x7f95400ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:51.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f954006bd80 con 0x7f95401516f0 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9540072870 con 0x7f9540151ad0 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9548000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f95400c1080 con 0x7f9540157c80 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9545600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 0x7f954014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9546000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9544c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9540151ad0 0x7f95400ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9545600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 0x7f954014d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36686/0 (socket says 172.21.15.132:36686) 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9546000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:52162/0 (socket says 172.21.15.132:52162) 2024-03-27T06:21:51.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.620+0000 7f9545600640 1 -- 172.21.15.132:0/1216505650 learned_addr learned my addr 172.21.15.132:0/1216505650 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 -- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9540151ad0 msgr2=0x7f95400ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9540151ad0 0x7f95400ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 -- 172.21.15.132:0/1216505650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 msgr2=0x7f954014d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 0x7f954014d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 -- 172.21.15.132:0/1216505650 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954014d760 con 0x7f9540157c80 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9546000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9538033000 tx=0x7f9538004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538044020 con 0x7f9540157c80 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f953803f470 con 0x7f9540157c80 2024-03-27T06:21:51.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95400ca7e0 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538039040 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.621+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f954014cd50 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.623+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9540068840 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.623+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9538047020 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.624+0000 7f953d600640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 0x7f952006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.624+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f95380ae3f0 con 0x7f9540157c80 2024-03-27T06:21:51.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.624+0000 7f9545600640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 0x7f952006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:51.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.625+0000 7f9545600640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 0x7f952006ea20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9534007b10 tx=0x7f9534007a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:51.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:51.628+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f953804e030 con 0x7f9540157c80 2024-03-27T06:21:52.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.053+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f954006e3a0 con 0x7f9540157c80 2024-03-27T06:21:52.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.054+0000 7f953d600640 1 -- 172.21.15.132:0/1216505650 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6062 (secure 0 0 0) 0x7f953807c0c0 con 0x7f9540157c80 2024-03-27T06:21:52.043 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:52.044 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:50.235819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:active","state_seq":36,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904,5284]},"id":1}]} 2024-03-27T06:21:52.044 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 41 2024-03-27T06:21:52.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.057+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 msgr2=0x7f952006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:52.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.057+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 0x7f952006ea20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9534007b10 tx=0x7f9534007a60 comp rx=0 tx=0).stop 2024-03-27T06:21:52.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.057+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 msgr2=0x7f95401f0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.057+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9538033000 tx=0x7f9538004a40 comp rx=0 tx=0).stop 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 shutdown_connections 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9540157c80 0x7f95401f0490 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f952006c560 0x7f952006ea20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9540151ad0 0x7f95400ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 --2- 172.21.15.132:0/1216505650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95401516f0 0x7f954014d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 >> 172.21.15.132:0/1216505650 conn(0x7f95400cc060 msgr2=0x7f95400c2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:52.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 shutdown_connections 2024-03-27T06:21:52.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:52.058+0000 7f9548000640 1 -- 172.21.15.132:0/1216505650 wait complete. 2024-03-27T06:21:52.072 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=4718 2024-03-27T06:21:52.072 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2024-03-27T06:21:57.073 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-03-27T06:21:57.073 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-27T06:21:57.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-27T06:21:57.081 INFO:tasks.ceph.mds.c:Started 2024-03-27T06:21:57.081 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:57.253 INFO:tasks.ceph.mds.c.smithi132.stdout:starting mds.c at 2024-03-27T06:21:57.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.276+0000 7fad5e200640 1 Processor -- start 2024-03-27T06:21:57.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.276+0000 7fad5e200640 1 -- start start 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad5e200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fad580c72c0 0x7fad5815b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad5e200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad580c7e00 con 0x7fad58072060 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad5e200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad580c80a0 con 0x7fad58154d80 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad5e200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fad580c9140 con 0x7fad580c72c0 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad57400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fad58154d80 0x7fad5806b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:51350/0 (socket says 172.21.15.132:51350) 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad57400640 1 -- 172.21.15.132:0/1705125149 learned_addr learned my addr 172.21.15.132:0/1705125149 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:21:57.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.277+0000 7fad56a00640 1 --2- 172.21.15.132:0/1705125149 >> v2:172.21.15.134:3300/0 conn(0x7fad580c72c0 0x7fad5815b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:57.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.278+0000 7fad56000640 1 -- 172.21.15.132:0/1705125149 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891434912 0 0) 0x7fad580c80a0 con 0x7fad58154d80 2024-03-27T06:21:57.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.278+0000 7fad56a00640 1 -- 172.21.15.132:0/1705125149 >> v1:172.21.15.134:6789/0 conn(0x7fad58154d80 legacy=0x7fad5806b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.278+0000 7fad56a00640 1 -- 172.21.15.132:0/1705125149 >> v1:172.21.15.132:6789/0 conn(0x7fad58072060 legacy=0x7fad58165210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.278+0000 7fad56a00640 1 -- 172.21.15.132:0/1705125149 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad580c8380 con 0x7fad580c72c0 2024-03-27T06:21:57.269 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.279+0000 7fad56a00640 1 --2- 172.21.15.132:0/1705125149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad5815b3b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad4402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fce568767ed0e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad56000640 1 -- 172.21.15.132:0/1705125149 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44034070 con 0x7fad580c72c0 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad56000640 1 -- 172.21.15.132:0/1705125149 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad44030610 con 0x7fad580c72c0 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad56000640 1 -- 172.21.15.132:0/1705125149 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad440309e0 con 0x7fad580c72c0 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad5e200640 1 -- 172.21.15.132:0/1705125149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 msgr2=0x7fad5815b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad5e200640 1 --2- 172.21.15.132:0/1705125149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad5815b3b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad4402b960 comp rx=0 tx=0).stop 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad5e200640 1 -- 172.21.15.132:0/1705125149 shutdown_connections 2024-03-27T06:21:57.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad5e200640 1 --2- 172.21.15.132:0/1705125149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad5815b3b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.280+0000 7fad5e200640 1 -- 172.21.15.132:0/1705125149 >> 172.21.15.132:0/1705125149 conn(0x7fad5806d7c0 msgr2=0x7fad5806dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:57.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.281+0000 7fad5e200640 1 -- 172.21.15.132:0/1705125149 shutdown_connections 2024-03-27T06:21:57.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.281+0000 7fad5e200640 1 -- 172.21.15.132:0/1705125149 wait complete. 2024-03-27T06:21:57.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.282+0000 7fad5e200640 1 Processor -- start 2024-03-27T06:21:57.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.282+0000 7fad5e200640 1 -- start start 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 0x7fad5815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad58154d80 0x7fad580cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad5815b8f0 con 0x7fad58072060 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fad58067360 con 0x7fad580c72c0 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad5e200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fad580c8940 con 0x7fad58154d80 2024-03-27T06:21:57.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad57400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 0x7fad5815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad56a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad56a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:46920/0 (socket says 172.21.15.132:46920) 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad56a00640 1 -- 172.21.15.132:0/1701990620 learned_addr learned my addr 172.21.15.132:0/1701990620 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.283+0000 7fad57e00640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad58154d80 0x7fad580cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 -- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad58154d80 msgr2=0x7fad580cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad58154d80 0x7fad580cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 -- 172.21.15.132:0/1701990620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 msgr2=0x7fad5815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 0x7fad5815c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 -- 172.21.15.132:0/1701990620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5815c9d0 con 0x7fad580c72c0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad57400640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 0x7fad5815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad56a00640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44034040 con 0x7fad580c72c0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad44004030 con 0x7fad580c72c0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad581f4780 con 0x7fad580c72c0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44004420 con 0x7fad580c72c0 2024-03-27T06:21:57.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.284+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad58159a00 con 0x7fad580c72c0 2024-03-27T06:21:57.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.286+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad58068840 con 0x7fad580c72c0 2024-03-27T06:21:57.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.287+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fad44037070 con 0x7fad580c72c0 2024-03-27T06:21:57.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.287+0000 7fad4f400640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 0x7fad3806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:57.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.288+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fad440ad490 con 0x7fad580c72c0 2024-03-27T06:21:57.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.288+0000 7fad57400640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 0x7fad3806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:57.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.288+0000 7fad57400640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 0x7fad3806ea20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad48000f60 tx=0x7fad48007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:57.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.295+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fad4407b3b0 con 0x7fad580c72c0 2024-03-27T06:21:57.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.527+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad5806c5c0 con 0x7fad580c72c0 2024-03-27T06:21:57.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.528+0000 7fad4f400640 1 -- 172.21.15.132:0/1701990620 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6062 (secure 0 0 0) 0x7fad4407f6b0 con 0x7fad580c72c0 2024-03-27T06:21:57.518 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:21:57.518 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:50.235819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:active","state_seq":36,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[4817,4904,5284]},"id":1}]} 2024-03-27T06:21:57.519 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 41 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 msgr2=0x7fad3806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 0x7fad3806ea20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad48000f60 tx=0x7fad48007e40 comp rx=0 tx=0).stop 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 msgr2=0x7fad581f4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad440046c0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 shutdown_connections 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fad58154d80 0x7fad580cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fad3806c560 0x7fad3806ea20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad580c72c0 0x7fad581f4240 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 --2- 172.21.15.132:0/1701990620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad58072060 0x7fad5815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 >> 172.21.15.132:0/1701990620 conn(0x7fad5806d7c0 msgr2=0x7fad581534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 shutdown_connections 2024-03-27T06:21:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:57.531+0000 7fad5e200640 1 -- 172.21.15.132:0/1701990620 wait complete. 2024-03-27T06:21:57.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-27T06:21:59.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:21:59.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.770+0000 7fb838c00640 1 Processor -- start 2024-03-27T06:21:59.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.771+0000 7fb838c00640 1 -- start start 2024-03-27T06:21:59.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb838c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb838c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb834151ad0 0x7fb8340c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb838c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb83406b520 con 0x7fb834073880 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb838c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fb834072be0 con 0x7fb834151ad0 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb838c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb83406b7c0 con 0x7fb8341516f0 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb82be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb834151ad0 0x7fb8340c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55620/0 (socket says 172.21.15.132:55620) 2024-03-27T06:21:59.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 -- 172.21.15.132:0/130527763 learned_addr learned my addr 172.21.15.132:0/130527763 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:59.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 -- 172.21.15.132:0/130527763 >> v2:172.21.15.134:3300/0 conn(0x7fb834151ad0 msgr2=0x7fb8340c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:21:59.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 --2- 172.21.15.132:0/130527763 >> v2:172.21.15.134:3300/0 conn(0x7fb834151ad0 0x7fb8340c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 -- 172.21.15.132:0/130527763 >> v1:172.21.15.132:6789/0 conn(0x7fb834073880 legacy=0x7fb834157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:59.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.773+0000 7fb831e00640 1 -- 172.21.15.132:0/130527763 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83406baa0 con 0x7fb8341516f0 2024-03-27T06:21:59.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.775+0000 7fb831e00640 1 --2- 172.21.15.132:0/130527763 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb82400bb10 tx=0x7fb82402e1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1f90cc7c38a7b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.775+0000 7fb831400640 1 -- 172.21.15.132:0/130527763 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82403b070 con 0x7fb8341516f0 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.775+0000 7fb831400640 1 -- 172.21.15.132:0/130527763 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb824036440 con 0x7fb8341516f0 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.775+0000 7fb831400640 1 -- 172.21.15.132:0/130527763 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824036810 con 0x7fb8341516f0 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 -- 172.21.15.132:0/130527763 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 msgr2=0x7fb8340c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 --2- 172.21.15.132:0/130527763 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb82400bb10 tx=0x7fb82402e1f0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 -- 172.21.15.132:0/130527763 shutdown_connections 2024-03-27T06:21:59.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 --2- 172.21.15.132:0/130527763 >> v2:172.21.15.134:3300/0 conn(0x7fb834151ad0 0x7fb8340c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 --2- 172.21.15.132:0/130527763 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb8341516f0 0x7fb8340c7e20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 -- 172.21.15.132:0/130527763 >> 172.21.15.132:0/130527763 conn(0x7fb8340cc060 msgr2=0x7fb8340cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:21:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 -- 172.21.15.132:0/130527763 shutdown_connections 2024-03-27T06:21:59.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.776+0000 7fb838c00640 1 -- 172.21.15.132:0/130527763 wait complete. 2024-03-27T06:21:59.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.778+0000 7fb838c00640 1 Processor -- start 2024-03-27T06:21:59.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.779+0000 7fb838c00640 1 -- start start 2024-03-27T06:21:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 0x7fb8341e2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 0x7fb8341f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 0x7fb8341f0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb83406bd80 con 0x7fb834073880 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb834072870 con 0x7fb8341516f0 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.780+0000 7fb838c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb8340c1080 con 0x7fb834151ad0 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb831e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 0x7fb8341e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 0x7fb8341f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb831e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 0x7fb8341e2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57002/0 (socket says 172.21.15.132:57002) 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb831e00640 1 -- 172.21.15.132:0/3411270751 learned_addr learned my addr 172.21.15.132:0/3411270751 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:21:59.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb82be00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 0x7fb8341f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 -- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 msgr2=0x7fb8341f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 0x7fb8341f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 -- 172.21.15.132:0/3411270751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 msgr2=0x7fb8341e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 0x7fb8341e2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 -- 172.21.15.132:0/3411270751 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8341e3650 con 0x7fb834151ad0 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb82be00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 0x7fb8341f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:21:59.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb832800640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 0x7fb8341f0800 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c690 tx=0x7fb82c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c003b70 con 0x7fb834151ad0 2024-03-27T06:21:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.781+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82c003d10 con 0x7fb834151ad0 2024-03-27T06:21:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.782+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c005310 con 0x7fb834151ad0 2024-03-27T06:21:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.782+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8341e2dd0 con 0x7fb834151ad0 2024-03-27T06:21:59.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.782+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8341f3cb0 con 0x7fb834151ad0 2024-03-27T06:21:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.783+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb834068840 con 0x7fb834151ad0 2024-03-27T06:21:59.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.784+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb82c0054b0 con 0x7fb834151ad0 2024-03-27T06:21:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.785+0000 7fb82a000640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 0x7fb81006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:21:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.785+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fb82c08b670 con 0x7fb834151ad0 2024-03-27T06:21:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.785+0000 7fb831e00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 0x7fb81006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:21:59.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.785+0000 7fb831e00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 0x7fb81006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb824039040 tx=0x7fb824008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:21:59.778 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:21:59.789+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb82c059ac0 con 0x7fb834151ad0 2024-03-27T06:22:00.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.026+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb83414b420 con 0x7fb834151ad0 2024-03-27T06:22:00.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.027+0000 7fb82a000640 1 -- 172.21.15.132:0/3411270751 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6890 (secure 0 0 0) 0x7fb82c0593c0 con 0x7fb834151ad0 2024-03-27T06:22:00.017 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:00.017 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5111,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1016263278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1016263278},{"type":"v1","addr":"172.21.15.132:6837","nonce":1016263278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:50.235819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:active","state_seq":36,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[5284,4904,4817]},"id":1}]} 2024-03-27T06:22:00.017 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 42 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 msgr2=0x7fb81006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 0x7fb81006e9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb824039040 tx=0x7fb824008040 comp rx=0 tx=0).stop 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 msgr2=0x7fb8341f0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 0x7fb8341f0800 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c690 tx=0x7fb82c00cb50 comp rx=0 tx=0).stop 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 shutdown_connections 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb834151ad0 0x7fb8341f0800 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb81006c530 0x7fb81006e9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8341516f0 0x7fb8341f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 --2- 172.21.15.132:0/3411270751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb834073880 0x7fb8341e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 >> 172.21.15.132:0/3411270751 conn(0x7fb8340cc060 msgr2=0x7fb8340c1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:00.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 shutdown_connections 2024-03-27T06:22:00.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.030+0000 7fb838c00640 1 -- 172.21.15.132:0/3411270751 wait complete. 2024-03-27T06:22:00.038 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-03-27T06:22:00.039 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-03-27T06:22:00.039 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-03-27T06:22:00.039 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 5 2024-03-27T06:22:00.039 INFO:tasks.mds_thrash:done joining 2024-03-27T06:22:00.040 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-27T06:22:00.060 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-27T06:22:00.061 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:22:00.061 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-27T06:22:00.111 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-27T06:22:00.111 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-27T06:22:00.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:22:00.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.416+0000 7f4044e00640 1 Processor -- start 2024-03-27T06:22:00.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.416+0000 7f4044e00640 1 -- start start 2024-03-27T06:22:00.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.417+0000 7f4044e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f404015b1f0 0x7f404006c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.417+0000 7f4044e00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4040071220 con 0x7f40401549c0 2024-03-27T06:22:00.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.417+0000 7f4044e00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404006d020 con 0x7f4040072210 2024-03-27T06:22:00.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.417+0000 7f4044e00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4040071d80 con 0x7f404015b1f0 2024-03-27T06:22:00.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403ea00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f404015b1f0 0x7f404006c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403fe00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4040072210 0x7f40401650d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:51364/0 (socket says 172.21.15.132:51364) 2024-03-27T06:22:00.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403fe00640 1 -- 172.21.15.132:0/1746861160 learned_addr learned my addr 172.21.15.132:0/1746861160 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155868271 0 0) 0x7f404006d020 con 0x7f4040072210 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4024003520 con 0x7f4040072210 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707486605 0 0) 0x7f4030001110 con 0x7f40401549c0 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.418+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f404006d020 con 0x7f40401549c0 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.419+0000 7f403ea00640 1 -- 172.21.15.132:0/1746861160 >> v1:172.21.15.134:6789/0 conn(0x7f4040072210 legacy=0x7f40401650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.419+0000 7f403ea00640 1 -- 172.21.15.132:0/1746861160 >> v1:172.21.15.132:6789/0 conn(0x7f40401549c0 legacy=0x7f404006ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.419+0000 7f403ea00640 1 -- 172.21.15.132:0/1746861160 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404006d300 con 0x7f404015b1f0 2024-03-27T06:22:00.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.420+0000 7f403ea00640 1 --2- 172.21.15.132:0/1746861160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f404015b1f0 0x7f404006c7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f40280097a0 tx=0x7f402802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f37e71b41955a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.420+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402802f810 con 0x7f404015b1f0 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.420+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402802f9b0 con 0x7f404015b1f0 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.420+0000 7f403e000640 1 -- 172.21.15.132:0/1746861160 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402802fd80 con 0x7f404015b1f0 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 -- 172.21.15.132:0/1746861160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f404015b1f0 msgr2=0x7f404006c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 --2- 172.21.15.132:0/1746861160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f404015b1f0 0x7f404006c7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f40280097a0 tx=0x7f402802b9f0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 -- 172.21.15.132:0/1746861160 shutdown_connections 2024-03-27T06:22:00.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 --2- 172.21.15.132:0/1746861160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f404015b1f0 0x7f404006c7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 -- 172.21.15.132:0/1746861160 >> 172.21.15.132:0/1746861160 conn(0x7f40400cc7a0 msgr2=0x7f40400c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:00.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 -- 172.21.15.132:0/1746861160 shutdown_connections 2024-03-27T06:22:00.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.421+0000 7f4044e00640 1 -- 172.21.15.132:0/1746861160 wait complete. 2024-03-27T06:22:00.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.423+0000 7f4044e00640 1 Processor -- start 2024-03-27T06:22:00.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.423+0000 7f4044e00640 1 -- start start 2024-03-27T06:22:00.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 0x7f40400ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 0x7f404015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f404015b1f0 0x7f404015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4040071a10 con 0x7f40401549c0 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4040071730 con 0x7f4040072210 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f4044e00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4040154da0 con 0x7f404015b1f0 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 0x7f404015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 0x7f40400ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f404015b1f0 0x7f404015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 0x7f404015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57012/0 (socket says 172.21.15.132:57012) 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403ea00640 1 -- 172.21.15.132:0/2406424756 learned_addr learned my addr 172.21.15.132:0/2406424756 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 -- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f404015b1f0 msgr2=0x7f404015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f404015b1f0 0x7f404015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 -- 172.21.15.132:0/2406424756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 msgr2=0x7f404015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 0x7f404015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.424+0000 7f403f400640 1 -- 172.21.15.132:0/2406424756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404015bee0 con 0x7f4040072210 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f403f400640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 0x7f40400ca570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f403000c3f0 tx=0x7f403000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40300052d0 con 0x7f4040072210 2024-03-27T06:22:00.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4030005490 con 0x7f4040072210 2024-03-27T06:22:00.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40401569d0 con 0x7f4040072210 2024-03-27T06:22:00.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030006c20 con 0x7f4040072210 2024-03-27T06:22:00.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.425+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4040156f90 con 0x7f4040072210 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.426+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40400690d0 con 0x7f4040072210 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.427+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4030007050 con 0x7f4040072210 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.428+0000 7f4037400640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 0x7f401c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.428+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f403008ad70 con 0x7f4040072210 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.428+0000 7f403ea00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 0x7f401c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.429+0000 7f403ea00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 0x7f401c06e9f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4040158150 tx=0x7f4028008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.436+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f40400690d0 con 0x7f4040072210 2024-03-27T06:22:00.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.635+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40400732c0 con 0x7f4040072210 2024-03-27T06:22:00.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.636+0000 7f4037400640 1 -- 172.21.15.132:0/2406424756 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17374 (secure 0 0 0) 0x7f40400732c0 con 0x7f4040072210 2024-03-27T06:22:00.626 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:00.627 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":44,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:21:29.204148+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":40,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6839/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6837/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6838/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6837/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.134:6836/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6836/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6834/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6835/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.132:6837/690619582":"2024-03-28T06:15:19.112063+0000","172.21.15.132:6835/690619582":"2024-03-28T06:15:19.112063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 msgr2=0x7f401c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 0x7f401c06e9f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4040158150 tx=0x7f4028008030 comp rx=0 tx=0).stop 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 msgr2=0x7f40400ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 0x7f40400ca570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f403000c3f0 tx=0x7f403000c8b0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 shutdown_connections 2024-03-27T06:22:00.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f404015b1f0 0x7f404015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40401549c0 0x7f404015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f401c06c530 0x7f401c06e9f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 --2- 172.21.15.132:0/2406424756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4040072210 0x7f40400ca570 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 >> 172.21.15.132:0/2406424756 conn(0x7f40400cc7a0 msgr2=0x7f40401542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 shutdown_connections 2024-03-27T06:22:00.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.639+0000 7f4044e00640 1 -- 172.21.15.132:0/2406424756 wait complete. 2024-03-27T06:22:00.647 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T06:13:56.429180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-27T06:14:00.506476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-27T06:14:01.952491+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 40, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-27T06:22:00.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-27T06:22:00.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.854+0000 7fa9a1c00640 1 Processor -- start 2024-03-27T06:22:00.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.854+0000 7fa9a1c00640 1 -- start start 2024-03-27T06:22:00.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa9a1c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa9a1c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa99c0d0740 0x7fa99c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa9a1c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c0ccd30 con 0x7fa99c154f80 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa9a1c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa99c155360 con 0x7fa99c0d0740 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa9a1c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa99c0ccfd0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99b400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa99c0d0740 0x7fa99c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99a000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55644/0 (socket says 172.21.15.132:55644) 2024-03-27T06:22:00.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99aa00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa99c154f80 0x7fa99c15b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54018/0 (socket says 172.21.15.132:54018) 2024-03-27T06:22:00.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99a000640 1 -- 172.21.15.132:0/714185839 learned_addr learned my addr 172.21.15.132:0/714185839 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:00.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99a000640 1 -- 172.21.15.132:0/714185839 >> v2:172.21.15.132:3300/0 conn(0x7fa99c0d0740 msgr2=0x7fa99c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.856+0000 7fa99a000640 1 --2- 172.21.15.132:0/714185839 >> v2:172.21.15.132:3300/0 conn(0x7fa99c0d0740 0x7fa99c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.857+0000 7fa99a000640 1 -- 172.21.15.132:0/714185839 >> v1:172.21.15.132:6789/0 conn(0x7fa99c154f80 legacy=0x7fa99c15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.857+0000 7fa99a000640 1 -- 172.21.15.132:0/714185839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c0cd2b0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa99a000640 1 --2- 172.21.15.132:0/714185839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a43d6b58e7cb69f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa999600640 1 -- 172.21.15.132:0/714185839 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988035020 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa999600640 1 -- 172.21.15.132:0/714185839 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa988031640 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa999600640 1 -- 172.21.15.132:0/714185839 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98802d040 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa9a1c00640 1 -- 172.21.15.132:0/714185839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 msgr2=0x7fa99c0d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.858+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/714185839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa988009870 tx=0x7fa98802b8f0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 -- 172.21.15.132:0/714185839 shutdown_connections 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/714185839 >> v2:172.21.15.132:3300/0 conn(0x7fa99c0d0740 0x7fa99c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/714185839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c0cb2f0 0x7fa99c0d0200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 -- 172.21.15.132:0/714185839 >> 172.21.15.132:0/714185839 conn(0x7fa99c06d9f0 msgr2=0x7fa99c06de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 -- 172.21.15.132:0/714185839 shutdown_connections 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.859+0000 7fa9a1c00640 1 -- 172.21.15.132:0/714185839 wait complete. 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.860+0000 7fa9a1c00640 1 Processor -- start 2024-03-27T06:22:00.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 -- start start 2024-03-27T06:22:00.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 0x7fa99c0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 0x7fa99c0b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c154f80 0x7fa99c0b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c0cc0a0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c0cd870 con 0x7fa99c0d0740 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.861+0000 7fa9a1c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa99c0cc220 con 0x7fa99c154f80 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 0x7fa99c0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 0x7fa99c0b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 0x7fa99c0b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:46970/0 (socket says 172.21.15.132:46970) 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99a000640 1 -- 172.21.15.132:0/920565913 learned_addr learned my addr 172.21.15.132:0/920565913 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99b400640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c154f80 0x7fa99c0b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 -- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c154f80 msgr2=0x7fa99c0b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c154f80 0x7fa99c0b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 -- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 msgr2=0x7fa99c0b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 0x7fa99c0b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 -- 172.21.15.132:0/920565913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c152ac0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.862+0000 7fa99aa00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 0x7fa99c0b3850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000f60 tx=0x7fa98c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.863+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0074e0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.863+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98c0047e0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.863+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c004bd0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.863+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c152500 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.863+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c0b3050 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.864+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99c0b6580 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.865+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa98c01a020 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.866+0000 7fa992a00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 0x7fa97c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:00.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.867+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fa98c08aec0 con 0x7fa99c0cb2f0 2024-03-27T06:22:00.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.867+0000 7fa99a000640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 0x7fa97c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:00.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.867+0000 7fa99a000640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 0x7fa97c06ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9880090b0 tx=0x7fa98803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:00.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:00.870+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa99c0b6580 con 0x7fa99c0cb2f0 2024-03-27T06:22:01.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.091+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa99c0cbe80 con 0x7fa99c0cb2f0 2024-03-27T06:22:01.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.092+0000 7fa992a00640 1 -- 172.21.15.132:0/920565913 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7fa98c058c20 con 0x7fa99c0cb2f0 2024-03-27T06:22:01.082 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-03-27T06:22:01.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.094+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 msgr2=0x7fa97c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.094+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 0x7fa97c06ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9880090b0 tx=0x7fa98803c040 comp rx=0 tx=0).stop 2024-03-27T06:22:01.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.094+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 msgr2=0x7fa99c0b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.094+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 0x7fa99c0b3850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000f60 tx=0x7fa98c00c760 comp rx=0 tx=0).stop 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 shutdown_connections 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa99c154f80 0x7fa99c0b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa97c06c560 0x7fa97c06ea20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa99c0d0740 0x7fa99c0b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 --2- 172.21.15.132:0/920565913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa99c0cb2f0 0x7fa99c0b3850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 >> 172.21.15.132:0/920565913 conn(0x7fa99c06d9f0 msgr2=0x7fa99c153790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:01.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 shutdown_connections 2024-03-27T06:22:01.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.095+0000 7fa9a1c00640 1 -- 172.21.15.132:0/920565913 wait complete. 2024-03-27T06:22:01.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-27T06:22:01.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.322+0000 7fb287c00640 1 Processor -- start 2024-03-27T06:22:01.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.322+0000 7fb287c00640 1 -- start start 2024-03-27T06:22:01.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb287c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb28015c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb287c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fb280152500 0x7fb2801529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb287c00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb280071930 con 0x7fb28015bfa0 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb287c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fb280072110 con 0x7fb280152500 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb287c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb28006c530 con 0x7fb28015c380 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb285200640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fb28015bfa0 0x7fb2800735f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54028/0 (socket says 172.21.15.132:54028) 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb285200640 1 -- 172.21.15.132:0/1844316577 learned_addr learned my addr 172.21.15.132:0/1844316577 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb285c00640 1 --2- 172.21.15.132:0/1844316577 >> v2:172.21.15.132:3300/0 conn(0x7fb280152500 0x7fb2801529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb27fe00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb28015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.324+0000 7fb27f400640 1 -- 172.21.15.132:0/1844316577 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461798642 0 0) 0x7fb280071930 con 0x7fb28015bfa0 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.325+0000 7fb285c00640 1 -- 172.21.15.132:0/1844316577 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 msgr2=0x7fb28015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.325+0000 7fb285c00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb28015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.325+0000 7fb285c00640 1 -- 172.21.15.132:0/1844316577 >> v1:172.21.15.132:6789/0 conn(0x7fb28015bfa0 legacy=0x7fb2800735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.325+0000 7fb285c00640 1 -- 172.21.15.132:0/1844316577 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb280071bd0 con 0x7fb280152500 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.326+0000 7fb285c00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2801529b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb2780097a0 tx=0x7fb27802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa88c5fc691193f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.326+0000 7fb27f400640 1 -- 172.21.15.132:0/1844316577 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278034070 con 0x7fb280152500 2024-03-27T06:22:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.326+0000 7fb27f400640 1 -- 172.21.15.132:0/1844316577 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb278030800 con 0x7fb280152500 2024-03-27T06:22:01.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.326+0000 7fb27f400640 1 -- 172.21.15.132:0/1844316577 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278030bd0 con 0x7fb280152500 2024-03-27T06:22:01.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 -- 172.21.15.132:0/1844316577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 msgr2=0x7fb2801529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2801529b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb2780097a0 tx=0x7fb27802b940 comp rx=0 tx=0).stop 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 -- 172.21.15.132:0/1844316577 shutdown_connections 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2801529b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 --2- 172.21.15.132:0/1844316577 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb28015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 -- 172.21.15.132:0/1844316577 >> 172.21.15.132:0/1844316577 conn(0x7fb2800c89a0 msgr2=0x7fb2800c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 -- 172.21.15.132:0/1844316577 shutdown_connections 2024-03-27T06:22:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.327+0000 7fb287c00640 1 -- 172.21.15.132:0/1844316577 wait complete. 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.329+0000 7fb287c00640 1 Processor -- start 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.329+0000 7fb287c00640 1 -- start start 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2800c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 0x7fb2800cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb2800b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb280071e70 con 0x7fb280152500 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb2800c97e0 con 0x7fb28015bfa0 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb287c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fb28006ca60 con 0x7fb28015c380 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb285200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2800c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb27fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 0x7fb2800cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb285c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb2800b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb27fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 0x7fb2800cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:46976/0 (socket says 172.21.15.132:46976) 2024-03-27T06:22:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb285c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb2800b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55676/0 (socket says 172.21.15.132:55676) 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.330+0000 7fb27fe00640 1 -- 172.21.15.132:0/1750738831 learned_addr learned my addr 172.21.15.132:0/1750738831 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 -- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 msgr2=0x7fb2800b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb2800b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 -- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 msgr2=0x7fb2800cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 0x7fb2800cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 -- 172.21.15.132:0/1750738831 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2801f0e40 con 0x7fb280152500 2024-03-27T06:22:01.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb285200640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2800c5530 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb27000c480 tx=0x7fb27000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270018070 con 0x7fb280152500 2024-03-27T06:22:01.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2700074e0 con 0x7fb280152500 2024-03-27T06:22:01.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2801f1120 con 0x7fb280152500 2024-03-27T06:22:01.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270004a20 con 0x7fb280152500 2024-03-27T06:22:01.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.331+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb28015b830 con 0x7fb280152500 2024-03-27T06:22:01.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.333+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb280072850 con 0x7fb280152500 2024-03-27T06:22:01.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.335+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fb27001b070 con 0x7fb280152500 2024-03-27T06:22:01.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.336+0000 7fb27cc00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 0x7fb26006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.336+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fb27008aaf0 con 0x7fb280152500 2024-03-27T06:22:01.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.336+0000 7fb27fe00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 0x7fb26006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.336+0000 7fb27fe00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 0x7fb26006e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb2800c5a70 tx=0x7fb26c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.341+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fb280072850 con 0x7fb280152500 2024-03-27T06:22:01.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.541+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2801528e0 con 0x7fb280152500 2024-03-27T06:22:01.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.542+0000 7fb27cc00640 1 -- 172.21.15.132:0/1750738831 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7fb2801528e0 con 0x7fb280152500 2024-03-27T06:22:01.532 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 32 2024-03-27T06:22:01.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 msgr2=0x7fb26006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 0x7fb26006e9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb2800c5a70 tx=0x7fb26c00b040 comp rx=0 tx=0).stop 2024-03-27T06:22:01.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 msgr2=0x7fb2800c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2800c5530 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb27000c480 tx=0x7fb27000c940 comp rx=0 tx=0).stop 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 shutdown_connections 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fb28015c380 0x7fb2800b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fb26006c530 0x7fb26006e9f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb28015bfa0 0x7fb2800cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 --2- 172.21.15.132:0/1750738831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb280152500 0x7fb2800c5530 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.544+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 >> 172.21.15.132:0/1750738831 conn(0x7fb2800c89a0 msgr2=0x7fb28006eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.545+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 shutdown_connections 2024-03-27T06:22:01.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.545+0000 7fb287c00640 1 -- 172.21.15.132:0/1750738831 wait complete. 2024-03-27T06:22:01.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-27T06:22:01.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.767+0000 7f6cf1c00640 1 Processor -- start 2024-03-27T06:22:01.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.767+0000 7f6cf1c00640 1 -- start start 2024-03-27T06:22:01.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.768+0000 7f6cf1c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 0x7f6cec0c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.768+0000 7f6cf1c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 0x7f6cec072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.768+0000 7f6cf1c00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cec06c530 con 0x7f6cec158360 2024-03-27T06:22:01.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6cf1c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f6cec073150 con 0x7f6cec151810 2024-03-27T06:22:01.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6cf1c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6cec06c7d0 con 0x7f6cec151bf0 2024-03-27T06:22:01.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ce3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 0x7f6cec072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ceb400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6cec158360 0x7f6cec072a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:51372/0 (socket says 172.21.15.132:51372) 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ceb400640 1 -- 172.21.15.132:0/758905283 learned_addr learned my addr 172.21.15.132:0/758905283 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ceaa00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 0x7f6cec0c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6cea000640 1 -- 172.21.15.132:0/758905283 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 23979819 0 0) 0x7f6cec06c530 con 0x7f6cec158360 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6cea000640 1 -- 172.21.15.132:0/758905283 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cd0003520 con 0x7f6cec158360 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ce3e00640 1 -- 172.21.15.132:0/758905283 >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 msgr2=0x7f6cec0c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ce3e00640 1 --2- 172.21.15.132:0/758905283 >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 0x7f6cec0c1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ce3e00640 1 -- 172.21.15.132:0/758905283 >> v1:172.21.15.134:6789/0 conn(0x7f6cec158360 legacy=0x7f6cec072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ce3e00640 1 -- 172.21.15.132:0/758905283 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec06cab0 con 0x7f6cec151bf0 2024-03-27T06:22:01.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.769+0000 7f6ceaa00640 1 --2- 172.21.15.132:0/758905283 >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 0x7f6cec0c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:01.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6ce3e00640 1 --2- 172.21.15.132:0/758905283 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 0x7f6cec072500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40097a0 tx=0x7f6cd402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1184a436705faee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6cea000640 1 -- 172.21.15.132:0/758905283 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4034070 con 0x7f6cec151bf0 2024-03-27T06:22:01.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6cea000640 1 -- 172.21.15.132:0/758905283 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd4030610 con 0x7f6cec151bf0 2024-03-27T06:22:01.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6cea000640 1 -- 172.21.15.132:0/758905283 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40309e0 con 0x7f6cec151bf0 2024-03-27T06:22:01.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6cf1c00640 1 -- 172.21.15.132:0/758905283 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 msgr2=0x7f6cec072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.771+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/758905283 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 0x7f6cec072500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40097a0 tx=0x7f6cd402b960 comp rx=0 tx=0).stop 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 -- 172.21.15.132:0/758905283 shutdown_connections 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/758905283 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec151bf0 0x7f6cec072500 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/758905283 >> v2:172.21.15.134:3300/0 conn(0x7f6cec151810 0x7f6cec0c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 -- 172.21.15.132:0/758905283 >> 172.21.15.132:0/758905283 conn(0x7f6cec0c89a0 msgr2=0x7f6cec0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 -- 172.21.15.132:0/758905283 shutdown_connections 2024-03-27T06:22:01.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.772+0000 7f6cf1c00640 1 -- 172.21.15.132:0/758905283 wait complete. 2024-03-27T06:22:01.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.773+0000 7f6cf1c00640 1 Processor -- start 2024-03-27T06:22:01.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.773+0000 7f6cf1c00640 1 -- start start 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cec151810 0x7f6cec1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cec154b00 0x7f6cec140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec157e40 con 0x7f6cec151810 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec157b60 con 0x7f6cec154b00 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.774+0000 7f6cf1c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f6cec0c22c0 con 0x7f6cec158360 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55684/0 (socket says 172.21.15.132:55684) 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 -- 172.21.15.132:0/3766123898 learned_addr learned my addr 172.21.15.132:0/3766123898 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ceaa00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cec151810 0x7f6cec1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ceb400640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cec154b00 0x7f6cec140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 -- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cec154b00 msgr2=0x7f6cec140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cec154b00 0x7f6cec140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 -- 172.21.15.132:0/3766123898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cec151810 msgr2=0x7f6cec1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cec151810 0x7f6cec1f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:01.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 -- 172.21.15.132:0/3766123898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec1f48f0 con 0x7f6cec158360 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.775+0000 7f6ce3e00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009890 tx=0x7f6cd40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.776+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4034040 con 0x7f6cec158360 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.776+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd4030410 con 0x7f6cec158360 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.776+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4006e10 con 0x7f6cec158360 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.776+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cec153510 con 0x7f6cec158360 2024-03-27T06:22:01.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.776+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cec150e90 con 0x7f6cec158360 2024-03-27T06:22:01.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.777+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cec0b2240 con 0x7f6cec158360 2024-03-27T06:22:01.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.778+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6cd4037070 con 0x7f6cec158360 2024-03-27T06:22:01.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.779+0000 7f6ce2a00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 0x7f6cc806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:01.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.779+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f6cd40ad6d0 con 0x7f6cec158360 2024-03-27T06:22:01.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.779+0000 7f6ceaa00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 0x7f6cc806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:01.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.780+0000 7f6ceaa00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 0x7f6cc806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009560 tx=0x7f6cdc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:01.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:01.783+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f6cec0b2240 con 0x7f6cec158360 2024-03-27T06:22:02.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.055+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6cec151060 con 0x7f6cec158360 2024-03-27T06:22:02.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.055+0000 7f6ce2a00640 1 -- 172.21.15.132:0/3766123898 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f6cd407b4c0 con 0x7f6cec158360 2024-03-27T06:22:02.045 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 64 2024-03-27T06:22:02.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 msgr2=0x7f6cc806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 0x7f6cc806e9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009560 tx=0x7f6cdc009190 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 msgr2=0x7f6cec151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009890 tx=0x7f6cd40046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 shutdown_connections 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f6cc806c530 0x7f6cc806e9f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cec154b00 0x7f6cec140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6cec158360 0x7f6cec151fe0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 --2- 172.21.15.132:0/3766123898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cec151810 0x7f6cec1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 >> 172.21.15.132:0/3766123898 conn(0x7f6cec0c89a0 msgr2=0x7f6cec14b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:02.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 shutdown_connections 2024-03-27T06:22:02.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.058+0000 7f6cf1c00640 1 -- 172.21.15.132:0/3766123898 wait complete. 2024-03-27T06:22:02.064 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-27T06:22:02.395 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.406+0000 7f7721800640 1 Processor -- start 2024-03-27T06:22:02.396 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.406+0000 7f7721800640 1 -- start start 2024-03-27T06:22:02.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.408+0000 7f7721800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f771c0c8360 0x7f771c0c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.408+0000 7f7721800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f771c06ca90 0x7f771c06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.408+0000 7f7721800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771c06da20 con 0x7f771c150970 2024-03-27T06:22:02.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.408+0000 7f7721800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f771c0c1e50 con 0x7f771c06ca90 2024-03-27T06:22:02.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.408+0000 7f7721800640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f771c06dce0 con 0x7f771c0c8360 2024-03-27T06:22:02.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.409+0000 7f771be00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f771c06ca90 0x7f771c06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:02.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.409+0000 7f771be00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f771c06ca90 0x7f771c06cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53278/0 (socket says 172.21.15.132:53278) 2024-03-27T06:22:02.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.409+0000 7f771be00640 1 -- 172.21.15.132:0/3697532341 learned_addr learned my addr 172.21.15.132:0/3697532341 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:02.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.410+0000 7f771be00640 1 -- 172.21.15.132:0/3697532341 >> v2:172.21.15.134:3300/0 conn(0x7f771c0c8360 msgr2=0x7f771c0c22e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T06:22:02.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.410+0000 7f771be00640 1 --2- 172.21.15.132:0/3697532341 >> v2:172.21.15.134:3300/0 conn(0x7f771c0c8360 0x7f771c0c22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.410+0000 7f771be00640 1 -- 172.21.15.132:0/3697532341 >> v1:172.21.15.134:6789/0 conn(0x7f771c150970 legacy=0x7f771c1571f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T06:22:02.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.410+0000 7f771be00640 1 -- 172.21.15.132:0/3697532341 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771c0c1520 con 0x7f771c06ca90 2024-03-27T06:22:02.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.412+0000 7f771be00640 1 --2- 172.21.15.132:0/3697532341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c06ca90 0x7f771c06cf20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f771c0c1fd0 tx=0x7f770c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04c6e27f279c620 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:02.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.414+0000 7f7719600640 1 -- 172.21.15.132:0/3697532341 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c035020 con 0x7f771c06ca90 2024-03-27T06:22:02.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.414+0000 7f7719600640 1 -- 172.21.15.132:0/3697532341 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c031890 con 0x7f771c06ca90 2024-03-27T06:22:02.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.414+0000 7f7719600640 1 -- 172.21.15.132:0/3697532341 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c031c60 con 0x7f771c06ca90 2024-03-27T06:22:02.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.416+0000 7f7721800640 1 -- 172.21.15.132:0/3697532341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c06ca90 msgr2=0x7f771c06cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.416+0000 7f7721800640 1 --2- 172.21.15.132:0/3697532341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c06ca90 0x7f771c06cf20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f771c0c1fd0 tx=0x7f770c02b930 comp rx=0 tx=0).stop 2024-03-27T06:22:02.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.416+0000 7f7721800640 1 -- 172.21.15.132:0/3697532341 shutdown_connections 2024-03-27T06:22:02.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.416+0000 7f7721800640 1 --2- 172.21.15.132:0/3697532341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c06ca90 0x7f771c06cf20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.416+0000 7f7721800640 1 --2- 172.21.15.132:0/3697532341 >> v2:172.21.15.134:3300/0 conn(0x7f771c0c8360 0x7f771c0c22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.417+0000 7f7721800640 1 -- 172.21.15.132:0/3697532341 >> 172.21.15.132:0/3697532341 conn(0x7f771c0c8910 msgr2=0x7f771c0c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:02.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.417+0000 7f7721800640 1 -- 172.21.15.132:0/3697532341 shutdown_connections 2024-03-27T06:22:02.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.417+0000 7f7721800640 1 -- 172.21.15.132:0/3697532341 wait complete. 2024-03-27T06:22:02.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.421+0000 7f7721800640 1 Processor -- start 2024-03-27T06:22:02.411 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.421+0000 7f7721800640 1 -- start start 2024-03-27T06:22:02.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 0x7f771c0cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f771c0c2820 con 0x7f771c0c8360 2024-03-27T06:22:02.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f771c0c1ae0 con 0x7f771c150970 2024-03-27T06:22:02.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.423+0000 7f7721800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f771c0c6810 con 0x7f771c06ca90 2024-03-27T06:22:02.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.424+0000 7f771be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:02.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.425+0000 7f771be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:48192/0 (socket says 172.21.15.132:48192) 2024-03-27T06:22:02.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.425+0000 7f771aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:02.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.425+0000 7f771aa00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35344/0 (socket says 172.21.15.132:35344) 2024-03-27T06:22:02.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.425+0000 7f771aa00640 1 -- 172.21.15.132:0/2774134941 learned_addr learned my addr 172.21.15.132:0/2774134941 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:02.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.425+0000 7f771a000640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 0x7f771c0cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:02.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.427+0000 7f771be00640 1 -- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 msgr2=0x7f771c1f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771be00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771be00640 1 -- 172.21.15.132:0/2774134941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 msgr2=0x7f771c0cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771be00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 0x7f771c0cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771be00640 1 -- 172.21.15.132:0/2774134941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771c1f2ad0 con 0x7f771c150970 2024-03-27T06:22:02.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771a000640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 0x7f771c0cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:02.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.428+0000 7f771aa00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:02.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.429+0000 7f771be00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f770c033000 tx=0x7f770c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:02.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.430+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c03ad60 con 0x7f771c150970 2024-03-27T06:22:02.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.430+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c0443f0 con 0x7f771c150970 2024-03-27T06:22:02.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.430+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c0447e0 con 0x7f771c150970 2024-03-27T06:22:02.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.432+0000 7f7721800640 1 -- 172.21.15.132:0/2774134941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771c14f0d0 con 0x7f771c150970 2024-03-27T06:22:02.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.433+0000 7f7721800640 1 -- 172.21.15.132:0/2774134941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771c1f1d90 con 0x7f771c150970 2024-03-27T06:22:02.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.438+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f770c04b060 con 0x7f771c150970 2024-03-27T06:22:02.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.438+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f0005290 con 0x7f771c150970 2024-03-27T06:22:02.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.441+0000 7f7712a00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 0x7f76fc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:02.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.442+0000 7f771aa00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 0x7f76fc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:02.433 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.443+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f770c079cf0 con 0x7f771c150970 2024-03-27T06:22:02.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.445+0000 7f771aa00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 0x7f76fc06ea20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f771400c550 tx=0x7f771400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:02.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.453+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f770c03f070 con 0x7f771c150970 2024-03-27T06:22:02.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.951+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f76f0005850 con 0x7f771c150970 2024-03-27T06:22:02.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.952+0000 7f7712a00640 1 -- 172.21.15.132:0/2774134941 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v44) ==== 81+0+618 (secure 0 0 0) 0x7f770c02d040 con 0x7f771c150970 2024-03-27T06:22:02.941 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.134:6839/3802035585 2024-03-28T06:19:09.174490+0000 2024-03-27T06:22:02.941 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.134:6837/3802035585 2024-03-28T06:19:09.174490+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.134:6838/2702381292 2024-03-28T06:18:09.157115+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6837/1424864911 2024-03-28T06:21:29.204044+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.134:6836/2702381292 2024-03-28T06:18:09.157115+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6836/3973814804 2024-03-28T06:16:39.129873+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6834/3973814804 2024-03-28T06:16:39.129873+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6835/1424864911 2024-03-28T06:21:29.204044+0000 2024-03-27T06:22:02.942 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6837/690619582 2024-03-28T06:15:19.112063+0000 2024-03-27T06:22:02.943 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132:6835/690619582 2024-03-28T06:15:19.112063+0000 2024-03-27T06:22:02.943 INFO:teuthology.orchestra.run.smithi132.stderr:listed 10 entries 2024-03-27T06:22:02.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.956+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 msgr2=0x7f76fc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.956+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 0x7f76fc06ea20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f771400c550 tx=0x7f771400b040 comp rx=0 tx=0).stop 2024-03-27T06:22:02.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.956+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 msgr2=0x7f771c0c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:02.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.956+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f770c033000 tx=0x7f770c0046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 shutdown_connections 2024-03-27T06:22:02.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f76fc06c560 0x7f76fc06ea20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f771c150970 0x7f771c0c3850 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f771c0c8360 0x7f771c0cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 --2- 172.21.15.132:0/2774134941 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f771c06ca90 0x7f771c1f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:02.947 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.957+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 >> 172.21.15.132:0/2774134941 conn(0x7f771c0c8910 msgr2=0x7f771c0c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:02.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.958+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 shutdown_connections 2024-03-27T06:22:02.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:02.958+0000 7f7707e00640 1 -- 172.21.15.132:0/2774134941 wait complete. 2024-03-27T06:22:02.978 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-27T06:22:02.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:22:02.979 DEBUG:teuthology.orchestra.run.smithi132:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-27T06:22:04.940 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:22:04.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:22:08.661 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:22:08.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:22:08.662 DEBUG:teuthology.orchestra.run.smithi132:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-27T06:22:08.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:22:08.725 DEBUG:teuthology.orchestra.run.smithi132:> ip netns list 2024-03-27T06:22:08.763 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-27T06:22:08.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:22:08.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-27T06:22:08.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-27T06:22:08.872 DEBUG:teuthology.orchestra.run.smithi132:> sudo ip link delete ceph-brx 2024-03-27T06:22:08.983 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-27T06:22:08.992 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T06:22:08.993 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:22:09.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.186+0000 7f1adc200640 1 Processor -- start 2024-03-27T06:22:09.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.186+0000 7f1adc200640 1 -- start start 2024-03-27T06:22:09.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.187+0000 7f1adc200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1ad40b11e0 0x7f1ad40c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1adc200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad415b790 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1adc200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad40c12d0 con 0x7f1ad415aa00 2024-03-27T06:22:09.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1adc200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad40c1080 con 0x7f1ad40b11e0 2024-03-27T06:22:09.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ad9800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1ad40b11e0 0x7f1ad40c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ada200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1ad415aa00 0x7f1ad415ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56964/0 (socket says 172.21.15.132:56964) 2024-03-27T06:22:09.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ada200640 1 -- 172.21.15.132:0/3202402124 learned_addr learned my addr 172.21.15.132:0/3202402124 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:09.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ad8e00640 1 --1- 172.21.15.132:0/3202402124 >> v1:172.21.15.132:6789/0 conn(0x7f1ad40c6fa0 0x7f1ad406b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:60452/0 (socket says 172.21.15.132:60452) 2024-03-27T06:22:09.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ad9800640 1 -- 172.21.15.132:0/3202402124 >> v1:172.21.15.134:6789/0 conn(0x7f1ad415aa00 legacy=0x7f1ad415ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ad9800640 1 -- 172.21.15.132:0/3202402124 >> v1:172.21.15.132:6789/0 conn(0x7f1ad40c6fa0 legacy=0x7f1ad406b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.188+0000 7f1ad9800640 1 -- 172.21.15.132:0/3202402124 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad40c15b0 con 0x7f1ad40b11e0 2024-03-27T06:22:09.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.190+0000 7f1ad9800640 1 --2- 172.21.15.132:0/3202402124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40b11e0 0x7f1ad40c6a60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=206788ebd8febb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.190+0000 7f1ad3e00640 1 -- 172.21.15.132:0/3202402124 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac802f810 con 0x7f1ad40b11e0 2024-03-27T06:22:09.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.190+0000 7f1ad3e00640 1 -- 172.21.15.132:0/3202402124 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac802f9b0 con 0x7f1ad40b11e0 2024-03-27T06:22:09.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.190+0000 7f1ad3e00640 1 -- 172.21.15.132:0/3202402124 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac802fd80 con 0x7f1ad40b11e0 2024-03-27T06:22:09.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 -- 172.21.15.132:0/3202402124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40b11e0 msgr2=0x7f1ad40c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 --2- 172.21.15.132:0/3202402124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40b11e0 0x7f1ad40c6a60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b9f0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 -- 172.21.15.132:0/3202402124 shutdown_connections 2024-03-27T06:22:09.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 --2- 172.21.15.132:0/3202402124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40b11e0 0x7f1ad40c6a60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 -- 172.21.15.132:0/3202402124 >> 172.21.15.132:0/3202402124 conn(0x7f1ad40cc060 msgr2=0x7f1ad40cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:09.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 -- 172.21.15.132:0/3202402124 shutdown_connections 2024-03-27T06:22:09.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.191+0000 7f1adc200640 1 -- 172.21.15.132:0/3202402124 wait complete. 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.193+0000 7f1adc200640 1 Processor -- start 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.193+0000 7f1adc200640 1 -- start start 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad40b11e0 0x7f1ad4071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1ad415aa00 0x7f1ad414ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40b15c0 con 0x7f1ad40b11e0 2024-03-27T06:22:09.185 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40c1c00 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1adc200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad40c2090 con 0x7f1ad415aa00 2024-03-27T06:22:09.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad9800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad40b11e0 0x7f1ad4071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:48206/0 (socket says 172.21.15.132:48206) 2024-03-27T06:22:09.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ada200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1ad415aa00 0x7f1ad414ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 -- 172.21.15.132:0/1413551421 learned_addr learned my addr 172.21.15.132:0/1413551421 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 -- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1ad415aa00 msgr2=0x7f1ad414ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1ad415aa00 0x7f1ad414ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 -- 172.21.15.132:0/1413551421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad40b11e0 msgr2=0x7f1ad4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.194+0000 7f1ad8e00640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad40b11e0 0x7f1ad4071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1ad8e00640 1 -- 172.21.15.132:0/1413551421 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad41f48a0 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1ad8e00640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000c430 tx=0x7f1ac000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac00074e0 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac0007680 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0004a00 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad41f4b30 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.195+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad4152300 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.196+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad4069560 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.197+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f1ac001a020 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.198+0000 7f1ad1600640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 0x7f1ab406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.198+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f1ac008afa0 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.198+0000 7f1ad9800640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 0x7f1ab406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.199+0000 7f1ad9800640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 0x7f1ab406e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.204+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f1ad4069560 con 0x7f1ad40c6fa0 2024-03-27T06:22:09.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.441+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ad41f4e10 con 0x7f1ab406c530 2024-03-27T06:22:09.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.455+0000 7f1ad1600640 1 -- 172.21.15.132:0/1413551421 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256707 (secure 0 0 0) 0x7f1ad41f4e10 con 0x7f1ab406c530 2024-03-27T06:22:09.448 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:09.449 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:22:09.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 msgr2=0x7f1ab406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 0x7f1ab406e9f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac8008030 comp rx=0 tx=0).stop 2024-03-27T06:22:09.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 msgr2=0x7f1ad414f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000c430 tx=0x7f1ac000c8f0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 shutdown_connections 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f1ad415aa00 0x7f1ad414ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f1ab406c530 0x7f1ab406e9f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ad40c6fa0 0x7f1ad414f9d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 --2- 172.21.15.132:0/1413551421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad40b11e0 0x7f1ad4071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 >> 172.21.15.132:0/1413551421 conn(0x7f1ad40cc060 msgr2=0x7f1ad41506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:09.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.461+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 shutdown_connections 2024-03-27T06:22:09.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.462+0000 7f1adc200640 1 -- 172.21.15.132:0/1413551421 wait complete. 2024-03-27T06:22:09.470 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-03-27T06:22:08.481172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123705486,"num_objects":10750,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10750,"num_whiteouts":0,"num_read":1825,"num_read_kb":514907,"num_write":25281,"num_write_kb":1250384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":91,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25139,"ondisk_log_size":25139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2131048,"kb_used_data":1755304,"kb_used_omap":12,"kb_used_meta":375667,"kb_avail":752843672,"statfs":{"total":773094113280,"available":770911920128,"internally_reserved":0,"allocated":1797431296,"data_stored":2247640716,"data_compressed":214703604,"data_compressed_allocated":244834304,"data_compressed_original":752640000,"omap_allocated":12751,"internal_metadata":384683569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":138,"apply_latency_ms":138,"commit_latency_ns":138000000,"apply_latency_ns":138000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":124519841,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":123664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002440"},"pg_stats":[{"pgid":"3.3f","version":"44'331","reported_seq":2434,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.926354+0000","last_change":"2024-03-27T06:18:40.841294+0000","last_active":"2024-03-27T06:22:02.926354+0000","last_peered":"2024-03-27T06:22:02.926354+0000","last_clean":"2024-03-27T06:22:02.926354+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:22:02.926354+0000","last_undegraded":"2024-03-27T06:22:02.926354+0000","last_fullsized":"2024-03-27T06:22:02.926354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:32:50.072917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048207270000000003,"stat_sum":{"num_bytes":26656138,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":18,"num_read_kb":8207,"num_write":331,"num_write_kb":26201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"44'314","reported_seq":2415,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.638786+0000","last_change":"2024-03-27T06:18:40.829515+0000","last_active":"2024-03-27T06:22:02.638786+0000","last_peered":"2024-03-27T06:22:02.638786+0000","last_clean":"2024-03-27T06:22:02.638786+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:22:02.638786+0000","last_undegraded":"2024-03-27T06:22:02.638786+0000","last_fullsized":"2024-03-27T06:22:02.638786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:20.543774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037350370000000001,"stat_sum":{"num_bytes":5169603,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":315,"num_write_kb":5186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"44'328","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.593021+0000","last_change":"2024-03-27T06:18:40.829898+0000","last_active":"2024-03-27T06:22:02.593021+0000","last_peered":"2024-03-27T06:22:02.593021+0000","last_clean":"2024-03-27T06:22:02.593021+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:22:02.593021+0000","last_undegraded":"2024-03-27T06:22:02.593021+0000","last_fullsized":"2024-03-27T06:22:02.593021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036092289999999998,"stat_sum":{"num_bytes":13816404,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":21,"num_read_kb":8211,"num_write":321,"num_write_kb":21844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"44'308","reported_seq":2407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.859463+0000","last_change":"2024-03-27T06:18:40.562894+0000","last_active":"2024-03-27T06:22:02.859463+0000","last_peered":"2024-03-27T06:22:02.859463+0000","last_clean":"2024-03-27T06:22:02.859463+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:22:02.859463+0000","last_undegraded":"2024-03-27T06:22:02.859463+0000","last_fullsized":"2024-03-27T06:22:02.859463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:29:51.024283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002773321,"stat_sum":{"num_bytes":19633039,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":13,"num_read_kb":8202,"num_write":307,"num_write_kb":19330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"44'350","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.930912+0000","last_change":"2024-03-27T06:18:40.840112+0000","last_active":"2024-03-27T06:22:02.930912+0000","last_peered":"2024-03-27T06:22:02.930912+0000","last_clean":"2024-03-27T06:22:02.930912+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:22:02.930912+0000","last_undegraded":"2024-03-27T06:22:02.930912+0000","last_fullsized":"2024-03-27T06:22:02.930912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:12:12.302147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071357900000000004,"stat_sum":{"num_bytes":5850441,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":350,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"44'320","reported_seq":2425,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.880517+0000","last_change":"2024-03-27T06:18:40.562623+0000","last_active":"2024-03-27T06:22:02.880517+0000","last_peered":"2024-03-27T06:22:02.880517+0000","last_clean":"2024-03-27T06:22:02.880517+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:22:02.880517+0000","last_undegraded":"2024-03-27T06:22:02.880517+0000","last_fullsized":"2024-03-27T06:22:02.880517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:33:31.576614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002466584,"stat_sum":{"num_bytes":14040535,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":20,"num_read_kb":3571,"num_write":321,"num_write_kb":13825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"44'351","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.806369+0000","last_change":"2024-03-27T06:18:40.838197+0000","last_active":"2024-03-27T06:22:02.806369+0000","last_peered":"2024-03-27T06:22:02.806369+0000","last_clean":"2024-03-27T06:22:02.806369+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:22:02.806369+0000","last_undegraded":"2024-03-27T06:22:02.806369+0000","last_fullsized":"2024-03-27T06:22:02.806369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:12:28.032906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056884500000000003,"stat_sum":{"num_bytes":24259091,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":18,"num_read_kb":4113,"num_write":354,"num_write_kb":23807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"44'323","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.745845+0000","last_change":"2024-03-27T06:18:40.830353+0000","last_active":"2024-03-27T06:22:02.745845+0000","last_peered":"2024-03-27T06:22:02.745845+0000","last_clean":"2024-03-27T06:22:02.745845+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:22:02.745845+0000","last_undegraded":"2024-03-27T06:22:02.745845+0000","last_fullsized":"2024-03-27T06:22:02.745845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033930689999999999,"stat_sum":{"num_bytes":15911530,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":12,"num_read_kb":4107,"num_write":326,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"44'320","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.877773+0000","last_change":"2024-03-27T06:18:40.834747+0000","last_active":"2024-03-27T06:22:02.877773+0000","last_peered":"2024-03-27T06:22:02.877773+0000","last_clean":"2024-03-27T06:22:02.877773+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:22:02.877773+0000","last_undegraded":"2024-03-27T06:22:02.877773+0000","last_fullsized":"2024-03-27T06:22:02.877773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:36:36.330080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054729009999999996,"stat_sum":{"num_bytes":6593503,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":20,"num_read_kb":4119,"num_write":318,"num_write_kb":10665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"44'308","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.744529+0000","last_change":"2024-03-27T06:18:40.841044+0000","last_active":"2024-03-27T06:22:02.744529+0000","last_peered":"2024-03-27T06:22:02.744529+0000","last_clean":"2024-03-27T06:22:02.744529+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:22:02.744529+0000","last_undegraded":"2024-03-27T06:22:02.744529+0000","last_fullsized":"2024-03-27T06:22:02.744529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:49.951013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068694350000000001,"stat_sum":{"num_bytes":18722866,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":20,"num_read_kb":9776,"num_write":307,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"44'298","reported_seq":2404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.851195+0000","last_change":"2024-03-27T06:18:40.840622+0000","last_active":"2024-03-27T06:22:02.851195+0000","last_peered":"2024-03-27T06:22:02.851195+0000","last_clean":"2024-03-27T06:22:02.851195+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:22:02.851195+0000","last_undegraded":"2024-03-27T06:22:02.851195+0000","last_fullsized":"2024-03-27T06:22:02.851195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:17:59.498179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070900370000000004,"stat_sum":{"num_bytes":18171100,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":21,"num_read_kb":8210,"num_write":297,"num_write_kb":17901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"44'297","reported_seq":2391,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.000855+0000","last_change":"2024-03-27T06:18:40.840141+0000","last_active":"2024-03-27T06:22:03.000855+0000","last_peered":"2024-03-27T06:22:03.000855+0000","last_clean":"2024-03-27T06:22:03.000855+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:22:03.000855+0000","last_undegraded":"2024-03-27T06:22:03.000855+0000","last_fullsized":"2024-03-27T06:22:03.000855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:41:13.001519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072900689999999997,"stat_sum":{"num_bytes":8281717,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":296,"num_write_kb":12341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"44'291","reported_seq":2396,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.646796+0000","last_change":"2024-03-27T06:18:40.826940+0000","last_active":"2024-03-27T06:22:02.646796+0000","last_peered":"2024-03-27T06:22:02.646796+0000","last_clean":"2024-03-27T06:22:02.646796+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:22:02.646796+0000","last_undegraded":"2024-03-27T06:22:02.646796+0000","last_fullsized":"2024-03-27T06:22:02.646796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029554939999999999,"stat_sum":{"num_bytes":19210972,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":21,"num_read_kb":4116,"num_write":287,"num_write_kb":27094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"44'335","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.553745+0000","last_change":"2024-03-27T06:18:40.827478+0000","last_active":"2024-03-27T06:22:02.553745+0000","last_peered":"2024-03-27T06:22:02.553745+0000","last_clean":"2024-03-27T06:22:02.553745+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:22:02.553745+0000","last_undegraded":"2024-03-27T06:22:02.553745+0000","last_fullsized":"2024-03-27T06:22:02.553745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:34:11.809253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003176613,"stat_sum":{"num_bytes":7395932,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":20,"num_read_kb":4114,"num_write":334,"num_write_kb":11493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.718682+0000","last_change":"2024-03-27T06:18:40.829238+0000","last_active":"2024-03-27T06:22:02.718682+0000","last_peered":"2024-03-27T06:22:02.718682+0000","last_clean":"2024-03-27T06:22:02.718682+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:22:02.718682+0000","last_undegraded":"2024-03-27T06:22:02.718682+0000","last_fullsized":"2024-03-27T06:22:02.718682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:21:38.019283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036973140000000002,"stat_sum":{"num_bytes":23111157,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":349,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.908232+0000","last_change":"2024-03-27T06:18:40.826025+0000","last_active":"2024-03-27T06:22:02.908232+0000","last_peered":"2024-03-27T06:22:02.908232+0000","last_clean":"2024-03-27T06:22:02.908232+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:22:02.908232+0000","last_undegraded":"2024-03-27T06:22:02.908232+0000","last_fullsized":"2024-03-27T06:22:02.908232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001289739,"stat_sum":{"num_bytes":12029261,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":347,"num_write_kb":15208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"44'339","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980167+0000","last_change":"2024-03-27T06:18:40.835662+0000","last_active":"2024-03-27T06:22:02.980167+0000","last_peered":"2024-03-27T06:22:02.980167+0000","last_clean":"2024-03-27T06:22:02.980167+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:22:02.980167+0000","last_undegraded":"2024-03-27T06:22:02.980167+0000","last_fullsized":"2024-03-27T06:22:02.980167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089932469999999994,"stat_sum":{"num_bytes":10346712,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":335,"num_write_kb":14376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"44'303","reported_seq":2414,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.702615+0000","last_change":"2024-03-27T06:18:40.826258+0000","last_active":"2024-03-27T06:22:02.702615+0000","last_peered":"2024-03-27T06:22:02.702615+0000","last_clean":"2024-03-27T06:22:02.702615+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:22:02.702615+0000","last_undegraded":"2024-03-27T06:22:02.702615+0000","last_fullsized":"2024-03-27T06:22:02.702615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:50:38.551582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010652260000000001,"stat_sum":{"num_bytes":8103937,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":302,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"44'319","reported_seq":2426,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.942674+0000","last_change":"2024-03-27T06:18:40.836497+0000","last_active":"2024-03-27T06:22:02.942674+0000","last_peered":"2024-03-27T06:22:02.942674+0000","last_clean":"2024-03-27T06:22:02.942674+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:22:02.942674+0000","last_undegraded":"2024-03-27T06:22:02.942674+0000","last_fullsized":"2024-03-27T06:22:02.942674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:43:03.085499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075917839999999999,"stat_sum":{"num_bytes":14133748,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":23,"num_read_kb":12737,"num_write":315,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"44'345","reported_seq":2453,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.001320+0000","last_change":"2024-03-27T06:18:40.840351+0000","last_active":"2024-03-27T06:22:03.001320+0000","last_peered":"2024-03-27T06:22:03.001320+0000","last_clean":"2024-03-27T06:22:03.001320+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:22:03.001320+0000","last_undegraded":"2024-03-27T06:22:03.001320+0000","last_fullsized":"2024-03-27T06:22:03.001320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:15:43.218527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072459589999999997,"stat_sum":{"num_bytes":23160385,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":24,"num_read_kb":12308,"num_write":342,"num_write_kb":30986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"44'335","reported_seq":2446,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.716055+0000","last_change":"2024-03-27T06:18:40.562416+0000","last_active":"2024-03-27T06:22:02.716055+0000","last_peered":"2024-03-27T06:22:02.716055+0000","last_clean":"2024-03-27T06:22:02.716055+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:22:02.716055+0000","last_undegraded":"2024-03-27T06:22:02.716055+0000","last_fullsized":"2024-03-27T06:22:02.716055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001666102,"stat_sum":{"num_bytes":14662751,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":26,"num_read_kb":4119,"num_write":336,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"44'327","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703008+0000","last_change":"2024-03-27T06:18:40.829281+0000","last_active":"2024-03-27T06:22:02.703008+0000","last_peered":"2024-03-27T06:22:02.703008+0000","last_clean":"2024-03-27T06:22:02.703008+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:22:02.703008+0000","last_undegraded":"2024-03-27T06:22:02.703008+0000","last_fullsized":"2024-03-27T06:22:02.703008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:26:03.545200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024688459999999998,"stat_sum":{"num_bytes":9427787,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":13475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"44'329","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.787322+0000","last_change":"2024-03-27T06:18:40.829763+0000","last_active":"2024-03-27T06:22:02.787322+0000","last_peered":"2024-03-27T06:22:02.787322+0000","last_clean":"2024-03-27T06:22:02.787322+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:22:02.787322+0000","last_undegraded":"2024-03-27T06:22:02.787322+0000","last_fullsized":"2024-03-27T06:22:02.787322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039259519999999999,"stat_sum":{"num_bytes":16419938,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":29,"num_read_kb":18059,"num_write":326,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"44'342","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.905364+0000","last_change":"2024-03-27T06:18:40.826230+0000","last_active":"2024-03-27T06:22:02.905364+0000","last_peered":"2024-03-27T06:22:02.905364+0000","last_clean":"2024-03-27T06:22:02.905364+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:22:02.905364+0000","last_undegraded":"2024-03-27T06:22:02.905364+0000","last_fullsized":"2024-03-27T06:22:02.905364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470573,"stat_sum":{"num_bytes":19403206,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":18,"num_read_kb":8208,"num_write":342,"num_write_kb":19130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"44'316","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730340+0000","last_change":"2024-03-27T06:18:40.829685+0000","last_active":"2024-03-27T06:22:02.730340+0000","last_peered":"2024-03-27T06:22:02.730340+0000","last_clean":"2024-03-27T06:22:02.730340+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:22:02.730340+0000","last_undegraded":"2024-03-27T06:22:02.730340+0000","last_fullsized":"2024-03-27T06:22:02.730340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035867099999999999,"stat_sum":{"num_bytes":14543064,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":314,"num_write_kb":14366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"44'367","reported_seq":2471,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980468+0000","last_change":"2024-03-27T06:18:40.562911+0000","last_active":"2024-03-27T06:22:02.980468+0000","last_peered":"2024-03-27T06:22:02.980468+0000","last_clean":"2024-03-27T06:22:02.980468+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:22:02.980468+0000","last_undegraded":"2024-03-27T06:22:02.980468+0000","last_fullsized":"2024-03-27T06:22:02.980468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:49:45.282319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002151956,"stat_sum":{"num_bytes":5854740,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":367,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"44'308","reported_seq":2411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.589889+0000","last_change":"2024-03-27T06:18:40.827317+0000","last_active":"2024-03-27T06:22:02.589889+0000","last_peered":"2024-03-27T06:22:02.589889+0000","last_clean":"2024-03-27T06:22:02.589889+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:22:02.589889+0000","last_undegraded":"2024-03-27T06:22:02.589889+0000","last_fullsized":"2024-03-27T06:22:02.589889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:28:40.555644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032490380000000001,"stat_sum":{"num_bytes":14027540,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":18,"num_read_kb":8206,"num_write":307,"num_write_kb":13857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"44'310","reported_seq":2410,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.743178+0000","last_change":"2024-03-27T06:18:40.827238+0000","last_active":"2024-03-27T06:22:02.743178+0000","last_peered":"2024-03-27T06:22:02.743178+0000","last_clean":"2024-03-27T06:22:02.743178+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:22:02.743178+0000","last_undegraded":"2024-03-27T06:22:02.743178+0000","last_fullsized":"2024-03-27T06:22:02.743178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:35:39.617870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003068571,"stat_sum":{"num_bytes":18001138,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":16,"num_read_kb":7383,"num_write":307,"num_write_kb":21016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"44'351","reported_seq":2458,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.617101+0000","last_change":"2024-03-27T06:18:40.829449+0000","last_active":"2024-03-27T06:22:02.617101+0000","last_peered":"2024-03-27T06:22:02.617101+0000","last_clean":"2024-03-27T06:22:02.617101+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:22:02.617101+0000","last_undegraded":"2024-03-27T06:22:02.617101+0000","last_fullsized":"2024-03-27T06:22:02.617101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:47:20.901338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036751610000000001,"stat_sum":{"num_bytes":22534890,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":22,"num_read_kb":12306,"num_write":351,"num_write_kb":22191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"44'351","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.911800+0000","last_change":"2024-03-27T06:18:40.836714+0000","last_active":"2024-03-27T06:22:02.911800+0000","last_peered":"2024-03-27T06:22:02.911800+0000","last_clean":"2024-03-27T06:22:02.911800+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:22:02.911800+0000","last_undegraded":"2024-03-27T06:22:02.911800+0000","last_fullsized":"2024-03-27T06:22:02.911800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044159999999999,"stat_sum":{"num_bytes":14250100,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":24,"num_read_kb":7593,"num_write":351,"num_write_kb":14104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"44'375","reported_seq":2479,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.663268+0000","last_change":"2024-03-27T06:18:40.825940+0000","last_active":"2024-03-27T06:22:02.663268+0000","last_peered":"2024-03-27T06:22:02.663268+0000","last_clean":"2024-03-27T06:22:02.663268+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:22:02.663268+0000","last_undegraded":"2024-03-27T06:22:02.663268+0000","last_fullsized":"2024-03-27T06:22:02.663268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:54:24.095203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011795639999999999,"stat_sum":{"num_bytes":10303903,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":19,"num_read_kb":1482,"num_write":377,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"44'371","reported_seq":2484,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.738005+0000","last_change":"2024-03-27T06:18:40.561817+0000","last_active":"2024-03-27T06:22:02.738005+0000","last_peered":"2024-03-27T06:22:02.738005+0000","last_clean":"2024-03-27T06:22:02.738005+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:22:02.738005+0000","last_undegraded":"2024-03-27T06:22:02.738005+0000","last_fullsized":"2024-03-27T06:22:02.738005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:57:04.359511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016911179999999999,"stat_sum":{"num_bytes":11109721,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":28,"num_read_kb":4123,"num_write":370,"num_write_kb":11044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"44'320","reported_seq":2423,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.653651+0000","last_change":"2024-03-27T06:18:40.831829+0000","last_active":"2024-03-27T06:22:02.653651+0000","last_peered":"2024-03-27T06:22:02.653651+0000","last_clean":"2024-03-27T06:22:02.653651+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:22:02.653651+0000","last_undegraded":"2024-03-27T06:22:02.653651+0000","last_fullsized":"2024-03-27T06:22:02.653651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:18:03.157765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057058689999999997,"stat_sum":{"num_bytes":18559363,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":18,"num_read_kb":4112,"num_write":320,"num_write_kb":18284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"43'17","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283493+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:21:30.283493+0000","last_peered":"2024-03-27T06:21:30.283493+0000","last_clean":"2024-03-27T06:21:30.283493+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:21:30.283493+0000","last_undegraded":"2024-03-27T06:21:30.283493+0000","last_fullsized":"2024-03-27T06:21:30.283493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195296,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":18,"num_write_kb":4248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'311","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.773731+0000","last_change":"2024-03-27T06:18:40.562829+0000","last_active":"2024-03-27T06:22:02.773731+0000","last_peered":"2024-03-27T06:22:02.773731+0000","last_clean":"2024-03-27T06:22:02.773731+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:22:02.773731+0000","last_undegraded":"2024-03-27T06:22:02.773731+0000","last_fullsized":"2024-03-27T06:22:02.773731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:17:17.674549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018454000000000001,"stat_sum":{"num_bytes":6743271,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":310,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"44'253","reported_seq":326,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.246658+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:21:55.246658+0000","last_peered":"2024-03-27T06:21:55.246658+0000","last_clean":"2024-03-27T06:21:55.246658+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:21:55.246658+0000","last_undegraded":"2024-03-27T06:21:55.246658+0000","last_fullsized":"2024-03-27T06:21:55.246658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":8577,"num_write":252,"num_write_kb":12472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"43'62","reported_seq":132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.244913+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:21:55.244913+0000","last_peered":"2024-03-27T06:21:55.244913+0000","last_clean":"2024-03-27T06:21:55.244913+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:21:55.244913+0000","last_undegraded":"2024-03-27T06:21:55.244913+0000","last_fullsized":"2024-03-27T06:21:55.244913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":63,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"44'399","reported_seq":2511,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.700284+0000","last_change":"2024-03-27T06:18:40.830513+0000","last_active":"2024-03-27T06:22:02.700284+0000","last_peered":"2024-03-27T06:22:02.700284+0000","last_clean":"2024-03-27T06:22:02.700284+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:22:02.700284+0000","last_undegraded":"2024-03-27T06:22:02.700284+0000","last_fullsized":"2024-03-27T06:22:02.700284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003103719,"stat_sum":{"num_bytes":18931276,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":28,"num_read_kb":12930,"num_write":396,"num_write_kb":26887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"44'377","reported_seq":442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:53.574559+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:21:53.574559+0000","last_peered":"2024-03-27T06:21:53.574559+0000","last_clean":"2024-03-27T06:21:53.574559+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:21:53.574559+0000","last_undegraded":"2024-03-27T06:21:53.574559+0000","last_fullsized":"2024-03-27T06:21:53.574559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":374,"num_write_kb":12507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"44'335","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.674076+0000","last_change":"2024-03-27T06:18:40.562924+0000","last_active":"2024-03-27T06:22:02.674076+0000","last_peered":"2024-03-27T06:22:02.674076+0000","last_clean":"2024-03-27T06:22:02.674076+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:22:02.674076+0000","last_undegraded":"2024-03-27T06:22:02.674076+0000","last_fullsized":"2024-03-27T06:22:02.674076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:17:24.783114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019163050000000001,"stat_sum":{"num_bytes":19664158,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":13845,"num_write":329,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"43'10","reported_seq":76,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.401768+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:21:30.401768+0000","last_peered":"2024-03-27T06:21:30.401768+0000","last_clean":"2024-03-27T06:21:30.401768+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:21:30.401768+0000","last_undegraded":"2024-03-27T06:21:30.401768+0000","last_fullsized":"2024-03-27T06:21:30.401768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"44'351","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.986843+0000","last_change":"2024-03-27T06:18:40.563693+0000","last_active":"2024-03-27T06:22:02.986843+0000","last_peered":"2024-03-27T06:22:02.986843+0000","last_clean":"2024-03-27T06:22:02.986843+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:22:02.986843+0000","last_undegraded":"2024-03-27T06:22:02.986843+0000","last_fullsized":"2024-03-27T06:22:02.986843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026075790000000001,"stat_sum":{"num_bytes":22946854,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":25,"num_read_kb":12308,"num_write":351,"num_write_kb":22593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"44'107","reported_seq":173,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.610595+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:21:29.610595+0000","last_peered":"2024-03-27T06:21:29.610595+0000","last_clean":"2024-03-27T06:21:29.610595+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:21:29.610595+0000","last_undegraded":"2024-03-27T06:21:29.610595+0000","last_fullsized":"2024-03-27T06:21:29.610595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":106,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"44'320","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.808658+0000","last_change":"2024-03-27T06:18:40.830168+0000","last_active":"2024-03-27T06:22:02.808658+0000","last_peered":"2024-03-27T06:22:02.808658+0000","last_clean":"2024-03-27T06:22:02.808658+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:22:02.808658+0000","last_undegraded":"2024-03-27T06:22:02.808658+0000","last_fullsized":"2024-03-27T06:22:02.808658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003650304,"stat_sum":{"num_bytes":10082281,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":22,"num_read_kb":4115,"num_write":319,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"43'280","reported_seq":346,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.210794+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:21:29.210794+0000","last_peered":"2024-03-27T06:21:29.210794+0000","last_clean":"2024-03-27T06:21:29.210794+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:21:29.210794+0000","last_undegraded":"2024-03-27T06:21:29.210794+0000","last_fullsized":"2024-03-27T06:21:29.210794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":280,"num_write_kb":4300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'315","reported_seq":2416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.750367+0000","last_change":"2024-03-27T06:18:40.835973+0000","last_active":"2024-03-27T06:22:02.750367+0000","last_peered":"2024-03-27T06:22:02.750367+0000","last_clean":"2024-03-27T06:22:02.750367+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:22:02.750367+0000","last_undegraded":"2024-03-27T06:22:02.750367+0000","last_fullsized":"2024-03-27T06:22:02.750367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084315699999999993,"stat_sum":{"num_bytes":18808790,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":315,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"44'127","reported_seq":197,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:48.532269+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:21:48.532269+0000","last_peered":"2024-03-27T06:21:48.532269+0000","last_clean":"2024-03-27T06:21:48.532269+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:21:48.532269+0000","last_undegraded":"2024-03-27T06:21:48.532269+0000","last_fullsized":"2024-03-27T06:21:48.532269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":8203,"num_write":123,"num_write_kb":12364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"44'327","reported_seq":2430,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.742161+0000","last_change":"2024-03-27T06:18:40.841177+0000","last_active":"2024-03-27T06:22:02.742161+0000","last_peered":"2024-03-27T06:22:02.742161+0000","last_clean":"2024-03-27T06:22:02.742161+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:22:02.742161+0000","last_undegraded":"2024-03-27T06:22:02.742161+0000","last_fullsized":"2024-03-27T06:22:02.742161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:44:07.500058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050386370000000003,"stat_sum":{"num_bytes":2468399,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":327,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"44'144","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.248730+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:21:55.248730+0000","last_peered":"2024-03-27T06:21:55.248730+0000","last_clean":"2024-03-27T06:21:55.248730+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:21:55.248730+0000","last_undegraded":"2024-03-27T06:21:55.248730+0000","last_fullsized":"2024-03-27T06:21:55.248730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":8224,"num_write":145,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'341","reported_seq":2450,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.612638+0000","last_change":"2024-03-27T06:18:40.826892+0000","last_active":"2024-03-27T06:22:02.612638+0000","last_peered":"2024-03-27T06:22:02.612638+0000","last_clean":"2024-03-27T06:22:02.612638+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:22:02.612638+0000","last_undegraded":"2024-03-27T06:22:02.612638+0000","last_fullsized":"2024-03-27T06:22:02.612638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002905866,"stat_sum":{"num_bytes":19666163,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":25,"num_read_kb":16403,"num_write":336,"num_write_kb":27577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"44'6","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.214987+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:21:29.214987+0000","last_peered":"2024-03-27T06:21:29.214987+0000","last_clean":"2024-03-27T06:21:29.214987+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:21:29.214987+0000","last_undegraded":"2024-03-27T06:21:29.214987+0000","last_fullsized":"2024-03-27T06:21:29.214987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"44'346","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.820209+0000","last_change":"2024-03-27T06:18:40.837185+0000","last_active":"2024-03-27T06:22:02.820209+0000","last_peered":"2024-03-27T06:22:02.820209+0000","last_clean":"2024-03-27T06:22:02.820209+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:22:02.820209+0000","last_undegraded":"2024-03-27T06:22:02.820209+0000","last_fullsized":"2024-03-27T06:22:02.820209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010165321999999999,"stat_sum":{"num_bytes":10617237,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":345,"num_write_kb":10547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"44'12","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981540+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:22:03.981540+0000","last_peered":"2024-03-27T06:22:03.981540+0000","last_clean":"2024-03-27T06:22:03.981540+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:22:03.981540+0000","last_undegraded":"2024-03-27T06:22:03.981540+0000","last_fullsized":"2024-03-27T06:22:03.981540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":11,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"44'347","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.588028+0000","last_change":"2024-03-27T06:18:40.831916+0000","last_active":"2024-03-27T06:22:02.588028+0000","last_peered":"2024-03-27T06:22:02.588028+0000","last_clean":"2024-03-27T06:22:02.588028+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:22:02.588028+0000","last_undegraded":"2024-03-27T06:22:02.588028+0000","last_fullsized":"2024-03-27T06:22:02.588028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:33:54.035813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054212890000000001,"stat_sum":{"num_bytes":24812922,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":17,"num_read_kb":8206,"num_write":342,"num_write_kb":26402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"44'278","reported_seq":359,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250589+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:21:55.250589+0000","last_peered":"2024-03-27T06:21:55.250589+0000","last_clean":"2024-03-27T06:21:55.250589+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:21:55.250589+0000","last_undegraded":"2024-03-27T06:21:55.250589+0000","last_fullsized":"2024-03-27T06:21:55.250589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":283,"num_write_kb":8391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"44'358","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.826841+0000","last_change":"2024-03-27T06:18:40.834701+0000","last_active":"2024-03-27T06:22:02.826841+0000","last_peered":"2024-03-27T06:22:02.826841+0000","last_clean":"2024-03-27T06:22:02.826841+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:22:02.826841+0000","last_undegraded":"2024-03-27T06:22:02.826841+0000","last_fullsized":"2024-03-27T06:22:02.826841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083622499999999999,"stat_sum":{"num_bytes":12063538,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":20,"num_read_kb":12305,"num_write":353,"num_write_kb":20111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"44'29","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212511+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:21:34.212511+0000","last_peered":"2024-03-27T06:21:34.212511+0000","last_clean":"2024-03-27T06:21:34.212511+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:21:34.212511+0000","last_undegraded":"2024-03-27T06:21:34.212511+0000","last_fullsized":"2024-03-27T06:21:34.212511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4103,"num_write":34,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'291","reported_seq":2389,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.989310+0000","last_change":"2024-03-27T06:18:40.563367+0000","last_active":"2024-03-27T06:22:02.989310+0000","last_peered":"2024-03-27T06:22:02.989310+0000","last_clean":"2024-03-27T06:22:02.989310+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:22:02.989310+0000","last_undegraded":"2024-03-27T06:22:02.989310+0000","last_fullsized":"2024-03-27T06:22:02.989310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022904700000000002,"stat_sum":{"num_bytes":11066744,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":12134,"num_write":288,"num_write_kb":15054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"44'28","reported_seq":93,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601016+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:21:29.601016+0000","last_peered":"2024-03-27T06:21:29.601016+0000","last_clean":"2024-03-27T06:21:29.601016+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:21:29.601016+0000","last_undegraded":"2024-03-27T06:21:29.601016+0000","last_fullsized":"2024-03-27T06:21:29.601016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":30,"num_write_kb":4134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"44'327","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.837377+0000","last_change":"2024-03-27T06:18:40.841709+0000","last_active":"2024-03-27T06:22:02.837377+0000","last_peered":"2024-03-27T06:22:02.837377+0000","last_clean":"2024-03-27T06:22:02.837377+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:22:02.837377+0000","last_undegraded":"2024-03-27T06:22:02.837377+0000","last_fullsized":"2024-03-27T06:22:02.837377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:43:54.568781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050322370000000002,"stat_sum":{"num_bytes":14587209,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":24,"num_read_kb":12307,"num_write":322,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"44'23","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283694+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:21:30.283694+0000","last_peered":"2024-03-27T06:21:30.283694+0000","last_clean":"2024-03-27T06:21:30.283694+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:21:30.283694+0000","last_undegraded":"2024-03-27T06:21:30.283694+0000","last_fullsized":"2024-03-27T06:21:30.283694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":23,"num_write_kb":4120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"44'356","reported_seq":2457,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.998863+0000","last_change":"2024-03-27T06:18:40.840461+0000","last_active":"2024-03-27T06:22:02.998863+0000","last_peered":"2024-03-27T06:22:02.998863+0000","last_clean":"2024-03-27T06:22:02.998863+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:22:02.998863+0000","last_undegraded":"2024-03-27T06:22:02.998863+0000","last_fullsized":"2024-03-27T06:22:02.998863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:35.295705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070052669999999999,"stat_sum":{"num_bytes":11205183,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":352,"num_write_kb":11129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"44'29","reported_seq":101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:58.564457+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:21:58.564457+0000","last_peered":"2024-03-27T06:21:58.564457+0000","last_clean":"2024-03-27T06:21:58.564457+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:21:58.564457+0000","last_undegraded":"2024-03-27T06:21:58.564457+0000","last_fullsized":"2024-03-27T06:21:58.564457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":32,"num_write_kb":4138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"44'323","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.968996+0000","last_change":"2024-03-27T06:18:40.835886+0000","last_active":"2024-03-27T06:22:02.968996+0000","last_peered":"2024-03-27T06:22:02.968996+0000","last_clean":"2024-03-27T06:22:02.968996+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:22:02.968996+0000","last_undegraded":"2024-03-27T06:22:02.968996+0000","last_fullsized":"2024-03-27T06:22:02.968996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:18:41.783945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086058899999999997,"stat_sum":{"num_bytes":14608246,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":19,"num_read_kb":4113,"num_write":322,"num_write_kb":18530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.384769+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:21:30.384769+0000","last_peered":"2024-03-27T06:21:30.384769+0000","last_clean":"2024-03-27T06:21:30.384769+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:21:30.384769+0000","last_undegraded":"2024-03-27T06:21:30.384769+0000","last_fullsized":"2024-03-27T06:21:30.384769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'84","reported_seq":152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212620+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:21:34.212620+0000","last_peered":"2024-03-27T06:21:34.212620+0000","last_clean":"2024-03-27T06:21:34.212620+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:21:34.212620+0000","last_undegraded":"2024-03-27T06:21:34.212620+0000","last_fullsized":"2024-03-27T06:21:34.212620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":86,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'321","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.778867+0000","last_change":"2024-03-27T06:18:40.563107+0000","last_active":"2024-03-27T06:22:02.778867+0000","last_peered":"2024-03-27T06:22:02.778867+0000","last_clean":"2024-03-27T06:22:02.778867+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:22:02.778867+0000","last_undegraded":"2024-03-27T06:22:02.778867+0000","last_fullsized":"2024-03-27T06:22:02.778867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:37:09.221434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002005691,"stat_sum":{"num_bytes":10118185,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":14,"num_read_kb":4109,"num_write":319,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"44'228","reported_seq":300,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.247793+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:21:55.247793+0000","last_peered":"2024-03-27T06:21:55.247793+0000","last_clean":"2024-03-27T06:21:55.247793+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:21:55.247793+0000","last_undegraded":"2024-03-27T06:21:55.247793+0000","last_fullsized":"2024-03-27T06:21:55.247793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":8210,"num_write":225,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'326","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.648394+0000","last_change":"2024-03-27T06:18:40.825563+0000","last_active":"2024-03-27T06:22:02.648394+0000","last_peered":"2024-03-27T06:22:02.648394+0000","last_clean":"2024-03-27T06:22:02.648394+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:22:02.648394+0000","last_undegraded":"2024-03-27T06:22:02.648394+0000","last_fullsized":"2024-03-27T06:22:02.648394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001779484,"stat_sum":{"num_bytes":10043455,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":324,"num_write_kb":9981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"44'90","reported_seq":241,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.244489+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:21:55.244489+0000","last_peered":"2024-03-27T06:21:55.244489+0000","last_clean":"2024-03-27T06:21:55.244489+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:21:55.244489+0000","last_undegraded":"2024-03-27T06:21:55.244489+0000","last_fullsized":"2024-03-27T06:21:55.244489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534287,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":595,"num_write":89,"num_write_kb":601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"44'332","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703670+0000","last_change":"2024-03-27T06:18:40.830316+0000","last_active":"2024-03-27T06:22:02.703670+0000","last_peered":"2024-03-27T06:22:02.703670+0000","last_clean":"2024-03-27T06:22:02.703670+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:22:02.703670+0000","last_undegraded":"2024-03-27T06:22:02.703670+0000","last_fullsized":"2024-03-27T06:22:02.703670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031179929999999999,"stat_sum":{"num_bytes":2779321,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":332,"num_write_kb":2894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"44'41","reported_seq":106,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.216718+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:21:34.216718+0000","last_peered":"2024-03-27T06:21:34.216718+0000","last_clean":"2024-03-27T06:21:34.216718+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:21:34.216718+0000","last_undegraded":"2024-03-27T06:21:34.216718+0000","last_fullsized":"2024-03-27T06:21:34.216718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":36,"num_write_kb":8227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"44'346","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.723176+0000","last_change":"2024-03-27T06:18:40.826852+0000","last_active":"2024-03-27T06:22:02.723176+0000","last_peered":"2024-03-27T06:22:02.723176+0000","last_clean":"2024-03-27T06:22:02.723176+0000","last_became_active":"2024-03-27T06:14:49.853879+0000","last_became_peered":"2024-03-27T06:14:49.853879+0000","last_unstale":"2024-03-27T06:22:02.723176+0000","last_undegraded":"2024-03-27T06:22:02.723176+0000","last_fullsized":"2024-03-27T06:22:02.723176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:54:30.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022092909999999999,"stat_sum":{"num_bytes":5308376,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":346,"num_write_kb":5365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"44'78","reported_seq":141,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:37.150979+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:21:37.150979+0000","last_peered":"2024-03-27T06:21:37.150979+0000","last_clean":"2024-03-27T06:21:37.150979+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:21:37.150979+0000","last_undegraded":"2024-03-27T06:21:37.150979+0000","last_fullsized":"2024-03-27T06:21:37.150979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":4150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"44'340","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.631346+0000","last_change":"2024-03-27T06:18:40.563208+0000","last_active":"2024-03-27T06:22:02.631346+0000","last_peered":"2024-03-27T06:22:02.631346+0000","last_clean":"2024-03-27T06:22:02.631346+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:22:02.631346+0000","last_undegraded":"2024-03-27T06:22:02.631346+0000","last_fullsized":"2024-03-27T06:22:02.631346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002460744,"stat_sum":{"num_bytes":18473602,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":23,"num_read_kb":12305,"num_write":340,"num_write_kb":22314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'587","reported_seq":652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:40.150130+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:21:40.150130+0000","last_peered":"2024-03-27T06:21:40.150130+0000","last_clean":"2024-03-27T06:21:40.150130+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:21:40.150130+0000","last_undegraded":"2024-03-27T06:21:40.150130+0000","last_fullsized":"2024-03-27T06:21:40.150130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":587,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"44'331","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.580275+0000","last_change":"2024-03-27T06:18:40.827659+0000","last_active":"2024-03-27T06:22:02.580275+0000","last_peered":"2024-03-27T06:22:02.580275+0000","last_clean":"2024-03-27T06:22:02.580275+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:22:02.580275+0000","last_undegraded":"2024-03-27T06:22:02.580275+0000","last_fullsized":"2024-03-27T06:22:02.580275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032966190000000002,"stat_sum":{"num_bytes":6064397,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":331,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"44'137","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.629820+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:22:03.629820+0000","last_peered":"2024-03-27T06:22:03.629820+0000","last_clean":"2024-03-27T06:22:03.629820+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:22:03.629820+0000","last_undegraded":"2024-03-27T06:22:03.629820+0000","last_fullsized":"2024-03-27T06:22:03.629820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":322,"num_write_kb":2012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"44'317","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.566427+0000","last_change":"2024-03-27T06:18:40.562700+0000","last_active":"2024-03-27T06:22:02.566427+0000","last_peered":"2024-03-27T06:22:02.566427+0000","last_clean":"2024-03-27T06:22:02.566427+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:22:02.566427+0000","last_undegraded":"2024-03-27T06:22:02.566427+0000","last_fullsized":"2024-03-27T06:22:02.566427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:57:15.843788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017608820000000001,"stat_sum":{"num_bytes":20225923,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":318,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"43'79","reported_seq":145,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.215719+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:21:29.215719+0000","last_peered":"2024-03-27T06:21:29.215719+0000","last_clean":"2024-03-27T06:21:29.215719+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:21:29.215719+0000","last_undegraded":"2024-03-27T06:21:29.215719+0000","last_fullsized":"2024-03-27T06:21:29.215719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":78,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'350","reported_seq":2469,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.556934+0000","last_change":"2024-03-27T06:18:40.562997+0000","last_active":"2024-03-27T06:22:02.556934+0000","last_peered":"2024-03-27T06:22:02.556934+0000","last_clean":"2024-03-27T06:22:02.556934+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:22:02.556934+0000","last_undegraded":"2024-03-27T06:22:02.556934+0000","last_fullsized":"2024-03-27T06:22:02.556934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:19:49.684960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018869889999999999,"stat_sum":{"num_bytes":32233430,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":35,"num_read_kb":18086,"num_write":345,"num_write_kb":31650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"44'31","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.609369+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:21:29.609369+0000","last_peered":"2024-03-27T06:21:29.609369+0000","last_clean":"2024-03-27T06:21:29.609369+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:21:29.609369+0000","last_undegraded":"2024-03-27T06:21:29.609369+0000","last_fullsized":"2024-03-27T06:21:29.609369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":35,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'347","reported_seq":2451,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.890096+0000","last_change":"2024-03-27T06:18:40.836137+0000","last_active":"2024-03-27T06:22:02.890096+0000","last_peered":"2024-03-27T06:22:02.890096+0000","last_clean":"2024-03-27T06:22:02.890096+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:22:02.890096+0000","last_undegraded":"2024-03-27T06:22:02.890096+0000","last_fullsized":"2024-03-27T06:22:02.890096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081323179999999995,"stat_sum":{"num_bytes":22949808,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":346,"num_write_kb":22595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"44'118","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601177+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:21:29.601177+0000","last_peered":"2024-03-27T06:21:29.601177+0000","last_clean":"2024-03-27T06:21:29.601177+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:21:29.601177+0000","last_undegraded":"2024-03-27T06:21:29.601177+0000","last_fullsized":"2024-03-27T06:21:29.601177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":120,"num_write_kb":4433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"44'355","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.550157+0000","last_change":"2024-03-27T06:18:40.562590+0000","last_active":"2024-03-27T06:22:02.550157+0000","last_peered":"2024-03-27T06:22:02.550157+0000","last_clean":"2024-03-27T06:22:02.550157+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:22:02.550157+0000","last_undegraded":"2024-03-27T06:22:02.550157+0000","last_fullsized":"2024-03-27T06:22:02.550157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:13:24.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001751964,"stat_sum":{"num_bytes":19413938,"num_objects":185,"num_object_clones":0,"num_object_copies":370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":21,"num_read_kb":12304,"num_write":357,"num_write_kb":23191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"44'26","reported_seq":91,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:33.720499+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:21:33.720499+0000","last_peered":"2024-03-27T06:21:33.720499+0000","last_clean":"2024-03-27T06:21:33.720499+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:21:33.720499+0000","last_undegraded":"2024-03-27T06:21:33.720499+0000","last_fullsized":"2024-03-27T06:21:33.720499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":4098,"num_write":30,"num_write_kb":4123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'319","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.572064+0000","last_change":"2024-03-27T06:18:40.826508+0000","last_active":"2024-03-27T06:22:02.572064+0000","last_peered":"2024-03-27T06:22:02.572064+0000","last_clean":"2024-03-27T06:22:02.572064+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:22:02.572064+0000","last_undegraded":"2024-03-27T06:22:02.572064+0000","last_fullsized":"2024-03-27T06:22:02.572064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:40.745990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001518585,"stat_sum":{"num_bytes":15553910,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":17,"num_read_kb":5089,"num_write":319,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"44'200","reported_seq":268,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981086+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:22:03.981086+0000","last_peered":"2024-03-27T06:22:03.981086+0000","last_clean":"2024-03-27T06:22:03.981086+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:22:03.981086+0000","last_undegraded":"2024-03-27T06:22:03.981086+0000","last_fullsized":"2024-03-27T06:22:03.981086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9715716,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":197,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'347","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.917806+0000","last_change":"2024-03-27T06:18:40.836916+0000","last_active":"2024-03-27T06:22:02.917806+0000","last_peered":"2024-03-27T06:22:02.917806+0000","last_clean":"2024-03-27T06:22:02.917806+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:22:02.917806+0000","last_undegraded":"2024-03-27T06:22:02.917806+0000","last_fullsized":"2024-03-27T06:22:02.917806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:26:09.736687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044719999999999,"stat_sum":{"num_bytes":10744528,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":347,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'307","reported_seq":2417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.975860+0000","last_change":"2024-03-27T06:18:40.835719+0000","last_active":"2024-03-27T06:22:02.975860+0000","last_peered":"2024-03-27T06:22:02.975860+0000","last_clean":"2024-03-27T06:22:02.975860+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:22:02.975860+0000","last_undegraded":"2024-03-27T06:22:02.975860+0000","last_fullsized":"2024-03-27T06:22:02.975860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086575760000000002,"stat_sum":{"num_bytes":8008268,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4119,"num_write":307,"num_write_kb":12040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"30'22","reported_seq":87,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.611265+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:21:29.611265+0000","last_peered":"2024-03-27T06:21:29.611265+0000","last_clean":"2024-03-27T06:21:29.611265+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:21:29.611265+0000","last_undegraded":"2024-03-27T06:21:29.611265+0000","last_fullsized":"2024-03-27T06:21:29.611265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":22,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"44'335","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.726938+0000","last_change":"2024-03-27T06:18:40.835158+0000","last_active":"2024-03-27T06:22:02.726938+0000","last_peered":"2024-03-27T06:22:02.726938+0000","last_clean":"2024-03-27T06:22:02.726938+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:22:02.726938+0000","last_undegraded":"2024-03-27T06:22:02.726938+0000","last_fullsized":"2024-03-27T06:22:02.726938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:10:12.399076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061971559999999997,"stat_sum":{"num_bytes":27398907,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":15,"num_read_kb":8205,"num_write":334,"num_write_kb":31029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"44'55","reported_seq":125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.239947+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:21:55.239947+0000","last_peered":"2024-03-27T06:21:55.239947+0000","last_clean":"2024-03-27T06:21:55.239947+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:21:55.239947+0000","last_undegraded":"2024-03-27T06:21:55.239947+0000","last_fullsized":"2024-03-27T06:21:55.239947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":12312,"num_write":46,"num_write_kb":12316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"44'312","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730202+0000","last_change":"2024-03-27T06:18:40.829954+0000","last_active":"2024-03-27T06:22:02.730202+0000","last_peered":"2024-03-27T06:22:02.730202+0000","last_clean":"2024-03-27T06:22:02.730202+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:22:02.730202+0000","last_undegraded":"2024-03-27T06:22:02.730202+0000","last_fullsized":"2024-03-27T06:22:02.730202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036426230000000002,"stat_sum":{"num_bytes":14422279,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":21,"num_read_kb":4923,"num_write":311,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"44'81","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978121+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:22:03.978121+0000","last_peered":"2024-03-27T06:22:03.978121+0000","last_clean":"2024-03-27T06:22:03.978121+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:22:03.978121+0000","last_undegraded":"2024-03-27T06:22:03.978121+0000","last_fullsized":"2024-03-27T06:22:03.978121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":75,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"44'342","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.795505+0000","last_change":"2024-03-27T06:18:40.830116+0000","last_active":"2024-03-27T06:22:02.795505+0000","last_peered":"2024-03-27T06:22:02.795505+0000","last_clean":"2024-03-27T06:22:02.795505+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:22:02.795505+0000","last_undegraded":"2024-03-27T06:22:02.795505+0000","last_fullsized":"2024-03-27T06:22:02.795505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036711629999999999,"stat_sum":{"num_bytes":15186793,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":8204,"num_write":343,"num_write_kb":14950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"44'129","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.980283+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:22:03.980283+0000","last_peered":"2024-03-27T06:22:03.980283+0000","last_clean":"2024-03-27T06:22:03.980283+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:22:03.980283+0000","last_undegraded":"2024-03-27T06:22:03.980283+0000","last_fullsized":"2024-03-27T06:22:03.980283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":4136,"num_write":129,"num_write_kb":8304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940893603,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":1256,"num_read_kb":394723,"num_write":21174,"num_write_kb":1064721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1712472064,"data_stored":1880247110,"data_compressed":164906076,"data_compressed_allocated":182468608,"data_compressed_original":405446656,"omap_allocated":0,"internal_metadata":0},"log_size":21254,"ondisk_log_size":21254,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182221515,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":453,"num_read_kb":120084,"num_write":3948,"num_write_kb":183247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":91,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":80830464,"data_stored":364443030,"data_compressed":49732790,"data_compressed_allocated":62275584,"data_compressed_original":346046464,"omap_allocated":0,"internal_metadata":0},"log_size":3768,"ondisk_log_size":3768,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170540,"kb_used_data":143524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94201300,"statfs":{"total":96636764160,"available":96462131200,"internally_reserved":0,"allocated":146968576,"data_stored":174700599,"data_compressed":11348325,"data_compressed_allocated":13275136,"data_compressed_original":46198784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":213540,"kb_used_data":169884,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94158300,"statfs":{"total":96636764160,"available":96418099200,"internally_reserved":0,"allocated":173961216,"data_stored":215859427,"data_compressed":22390241,"data_compressed_allocated":25247744,"data_compressed_original":73641984,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291908,"kb_used_data":242684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94079932,"statfs":{"total":96636764160,"available":96337850368,"internally_reserved":0,"allocated":248508416,"data_stored":304860446,"data_compressed":28268738,"data_compressed_allocated":32043008,"data_compressed_original":96636928,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266772,"kb_used_data":217548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94105068,"statfs":{"total":96636764160,"available":96363589632,"internally_reserved":0,"allocated":222769152,"data_stored":282203639,"data_compressed":27135224,"data_compressed_allocated":30642176,"data_compressed_original":95821824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313816,"kb_used_data":264592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94058024,"statfs":{"total":96636764160,"available":96315416576,"internally_reserved":0,"allocated":270942208,"data_stored":358995823,"data_compressed":39540899,"data_compressed_allocated":45490176,"data_compressed_original":142065664,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267308,"kb_used_data":222244,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94104532,"statfs":{"total":96636764160,"available":96363040768,"internally_reserved":0,"allocated":227577856,"data_stored":275207385,"data_compressed":26559283,"data_compressed_allocated":30052352,"data_compressed_original":85663744,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246944,"kb_used_data":201880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94124896,"statfs":{"total":96636764160,"available":96383893504,"internally_reserved":0,"allocated":206725120,"data_stored":255586545,"data_compressed":23288424,"data_compressed_allocated":26611712,"data_compressed_original":81715200,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360220,"kb_used_data":292948,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94011620,"statfs":{"total":96636764160,"available":96267898880,"internally_reserved":0,"allocated":299978752,"data_stored":380226852,"data_compressed":36172470,"data_compressed_allocated":41472000,"data_compressed_original":130895872,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":67110666,"data_compressed":8915435,"data_compressed_allocated":11333632,"data_compressed_original":64311296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7794688,"data_stored":37748896,"data_compressed":4858381,"data_compressed_allocated":6283264,"data_compressed_original":36245504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6713344,"data_stored":30687942,"data_compressed":4449598,"data_compressed_allocated":5500928,"data_compressed_original":29491200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19513344,"data_stored":81020596,"data_compressed":11156334,"data_compressed_allocated":14245888,"data_compressed_original":75771904,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":50333888,"data_compressed":7205527,"data_compressed_allocated":8712192,"data_compressed_original":46735360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8495104,"data_stored":42478411,"data_compressed":5618062,"data_compressed_allocated":6979584,"data_compressed_original":40984576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5763072,"data_stored":29361702,"data_compressed":4140996,"data_compressed_allocated":5042176,"data_compressed_original":28651520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":25700929,"data_compressed":3388457,"data_compressed_allocated":4177920,"data_compressed_original":23855104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285327360,"data_stored":312894956,"data_compressed":27257035,"data_compressed_allocated":30138368,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198365184,"data_stored":217026051,"data_compressed":18397674,"data_compressed_allocated":20283392,"data_compressed_original":44896256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220364800,"data_stored":244298213,"data_compressed":22109685,"data_compressed_allocated":24551424,"data_compressed_original":56172544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250929152,"data_stored":277753997,"data_compressed":28384565,"data_compressed_allocated":31244288,"data_compressed_original":66293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209920000,"data_stored":231648521,"data_compressed":19929697,"data_compressed_allocated":21929984,"data_compressed_original":49086464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239448064,"data_stored":261570437,"data_compressed":22618307,"data_compressed_allocated":25018368,"data_compressed_original":55078912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167698432,"data_stored":186276495,"data_compressed":18249245,"data_compressed_allocated":20205568,"data_compressed_original":44990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140419072,"data_stored":148778440,"data_compressed":7959868,"data_compressed_allocated":9097216,"data_compressed_original":22343680,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:22:09.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:22:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.729+0000 7f7a93200640 1 Processor -- start 2024-03-27T06:22:09.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.729+0000 7f7a93200640 1 -- start start 2024-03-27T06:22:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a93200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7a8c0ca160 0x7f7a8c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a93200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f7a8c0cb5b0 0x7f7a8c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a93200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a8c154fe0 con 0x7f7a8c0710e0 2024-03-27T06:22:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a93200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8c0caa80 con 0x7f7a8c0cb5b0 2024-03-27T06:22:09.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a93200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8c0ca540 con 0x7f7a8c0ca160 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a8be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7a8c0ca160 0x7f7a8c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a91200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7a8c0710e0 0x7f7a8c06cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:56968/0 (socket says 172.21.15.132:56968) 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a91200640 1 -- 172.21.15.132:0/189330015 learned_addr learned my addr 172.21.15.132:0/189330015 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a8b400640 1 --2- 172.21.15.132:0/189330015 >> v2:172.21.15.132:3300/0 conn(0x7f7a8c0cb5b0 0x7f7a8c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a8be00640 1 -- 172.21.15.132:0/189330015 >> v2:172.21.15.132:3300/0 conn(0x7f7a8c0cb5b0 msgr2=0x7f7a8c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:09.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a8be00640 1 --2- 172.21.15.132:0/189330015 >> v2:172.21.15.132:3300/0 conn(0x7f7a8c0cb5b0 0x7f7a8c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.731+0000 7f7a8be00640 1 -- 172.21.15.132:0/189330015 >> v1:172.21.15.134:6789/0 conn(0x7f7a8c0710e0 legacy=0x7f7a8c06cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.732+0000 7f7a8be00640 1 -- 172.21.15.132:0/189330015 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c0cdfd0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.733+0000 7f7a8be00640 1 --2- 172.21.15.132:0/189330015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0ca160 0x7f7a8c15b320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0719a0 tx=0x7f7a7c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7bbe0ad404a844f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.733+0000 7f7a8aa00640 1 -- 172.21.15.132:0/189330015 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c035020 con 0x7f7a8c0ca160 2024-03-27T06:22:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.733+0000 7f7a8aa00640 1 -- 172.21.15.132:0/189330015 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7c031830 con 0x7f7a8c0ca160 2024-03-27T06:22:09.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.733+0000 7f7a8aa00640 1 -- 172.21.15.132:0/189330015 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031c00 con 0x7f7a8c0ca160 2024-03-27T06:22:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 -- 172.21.15.132:0/189330015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0ca160 msgr2=0x7f7a8c15b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 --2- 172.21.15.132:0/189330015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0ca160 0x7f7a8c15b320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0719a0 tx=0x7f7a7c02b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 -- 172.21.15.132:0/189330015 shutdown_connections 2024-03-27T06:22:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 --2- 172.21.15.132:0/189330015 >> v2:172.21.15.132:3300/0 conn(0x7f7a8c0cb5b0 0x7f7a8c070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 --2- 172.21.15.132:0/189330015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0ca160 0x7f7a8c15b320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 -- 172.21.15.132:0/189330015 >> 172.21.15.132:0/189330015 conn(0x7f7a8c073120 msgr2=0x7f7a8c073500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:09.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 -- 172.21.15.132:0/189330015 shutdown_connections 2024-03-27T06:22:09.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.734+0000 7f7a93200640 1 -- 172.21.15.132:0/189330015 wait complete. 2024-03-27T06:22:09.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.736+0000 7f7a93200640 1 Processor -- start 2024-03-27T06:22:09.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.737+0000 7f7a93200640 1 -- start start 2024-03-27T06:22:09.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 0x7f7a8c1f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 0x7f7a8c06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c0ce850 con 0x7f7a8c0710e0 2024-03-27T06:22:09.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c0ce570 con 0x7f7a8c0cb5b0 2024-03-27T06:22:09.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.738+0000 7f7a93200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c0ceb30 con 0x7f7a8c0ca160 2024-03-27T06:22:09.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 0x7f7a8c1f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35360/0 (socket says 172.21.15.132:35360) 2024-03-27T06:22:09.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 -- 172.21.15.132:0/969875656 learned_addr learned my addr 172.21.15.132:0/969875656 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a91200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 0x7f7a8c06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 -- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 msgr2=0x7f7a8c06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 0x7f7a8c06cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 -- 172.21.15.132:0/969875656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 msgr2=0x7f7a8c1f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 0x7f7a8c1f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 -- 172.21.15.132:0/969875656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c1f49e0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8be00640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 0x7f7a8c1f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:09.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a91200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 0x7f7a8c06cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:09.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.739+0000 7f7a8b400640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e9b0 tx=0x7f7a7800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.740+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78017070 con 0x7f7a8c0ca160 2024-03-27T06:22:09.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.740+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7800c9a0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.740+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7800cd90 con 0x7f7a8c0ca160 2024-03-27T06:22:09.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.740+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c1f14a0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.740+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c0cd0f0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.741+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c068840 con 0x7f7a8c0ca160 2024-03-27T06:22:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.742+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f7a7801a070 con 0x7f7a8c0ca160 2024-03-27T06:22:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.742+0000 7f7a83e00640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 0x7f7a6c06ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.742+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f7a7808b890 con 0x7f7a8c0ca160 2024-03-27T06:22:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.743+0000 7f7a8be00640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 0x7f7a6c06ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:09.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.743+0000 7f7a8be00640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 0x7f7a6c06ea70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:09.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.748+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f7a780595e0 con 0x7f7a8c0ca160 2024-03-27T06:22:09.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.944+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7a8c153750 con 0x7f7a6c06c5b0 2024-03-27T06:22:09.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.956+0000 7f7a83e00640 1 -- 172.21.15.132:0/969875656 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256707 (secure 0 0 0) 0x7f7a7c086ae0 con 0x7f7a6c06c5b0 2024-03-27T06:22:09.950 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:09.951 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:22:09.953 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 msgr2=0x7f7a6c06ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 0x7f7a6c06ea70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c03c040 comp rx=0 tx=0).stop 2024-03-27T06:22:09.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 msgr2=0x7f7a8c1f0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:09.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e9b0 tx=0x7f7a7800ee70 comp rx=0 tx=0).stop 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 shutdown_connections 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f7a6c06c5b0 0x7f7a6c06ea70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a8c0cb5b0 0x7f7a8c06cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f7a8c0ca160 0x7f7a8c1f0f60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 --2- 172.21.15.132:0/969875656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a8c0710e0 0x7f7a8c1f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 >> 172.21.15.132:0/969875656 conn(0x7f7a8c073120 msgr2=0x7f7a8c153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:09.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 shutdown_connections 2024-03-27T06:22:09.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:09.961+0000 7f7a93200640 1 -- 172.21.15.132:0/969875656 wait complete. 2024-03-27T06:22:09.968 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-03-27T06:22:08.481172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123705486,"num_objects":10750,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10750,"num_whiteouts":0,"num_read":1825,"num_read_kb":514907,"num_write":25281,"num_write_kb":1250384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":91,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25139,"ondisk_log_size":25139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2131048,"kb_used_data":1755304,"kb_used_omap":12,"kb_used_meta":375667,"kb_avail":752843672,"statfs":{"total":773094113280,"available":770911920128,"internally_reserved":0,"allocated":1797431296,"data_stored":2247640716,"data_compressed":214703604,"data_compressed_allocated":244834304,"data_compressed_original":752640000,"omap_allocated":12751,"internal_metadata":384683569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":138,"apply_latency_ms":138,"commit_latency_ns":138000000,"apply_latency_ns":138000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":124519841,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":123664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002440"},"pg_stats":[{"pgid":"3.3f","version":"44'331","reported_seq":2434,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.926354+0000","last_change":"2024-03-27T06:18:40.841294+0000","last_active":"2024-03-27T06:22:02.926354+0000","last_peered":"2024-03-27T06:22:02.926354+0000","last_clean":"2024-03-27T06:22:02.926354+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:22:02.926354+0000","last_undegraded":"2024-03-27T06:22:02.926354+0000","last_fullsized":"2024-03-27T06:22:02.926354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:32:50.072917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048207270000000003,"stat_sum":{"num_bytes":26656138,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":18,"num_read_kb":8207,"num_write":331,"num_write_kb":26201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"44'314","reported_seq":2415,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.638786+0000","last_change":"2024-03-27T06:18:40.829515+0000","last_active":"2024-03-27T06:22:02.638786+0000","last_peered":"2024-03-27T06:22:02.638786+0000","last_clean":"2024-03-27T06:22:02.638786+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:22:02.638786+0000","last_undegraded":"2024-03-27T06:22:02.638786+0000","last_fullsized":"2024-03-27T06:22:02.638786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:20.543774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037350370000000001,"stat_sum":{"num_bytes":5169603,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":315,"num_write_kb":5186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"44'328","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.593021+0000","last_change":"2024-03-27T06:18:40.829898+0000","last_active":"2024-03-27T06:22:02.593021+0000","last_peered":"2024-03-27T06:22:02.593021+0000","last_clean":"2024-03-27T06:22:02.593021+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:22:02.593021+0000","last_undegraded":"2024-03-27T06:22:02.593021+0000","last_fullsized":"2024-03-27T06:22:02.593021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036092289999999998,"stat_sum":{"num_bytes":13816404,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":21,"num_read_kb":8211,"num_write":321,"num_write_kb":21844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"44'308","reported_seq":2407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.859463+0000","last_change":"2024-03-27T06:18:40.562894+0000","last_active":"2024-03-27T06:22:02.859463+0000","last_peered":"2024-03-27T06:22:02.859463+0000","last_clean":"2024-03-27T06:22:02.859463+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:22:02.859463+0000","last_undegraded":"2024-03-27T06:22:02.859463+0000","last_fullsized":"2024-03-27T06:22:02.859463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:29:51.024283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002773321,"stat_sum":{"num_bytes":19633039,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":13,"num_read_kb":8202,"num_write":307,"num_write_kb":19330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"44'350","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.930912+0000","last_change":"2024-03-27T06:18:40.840112+0000","last_active":"2024-03-27T06:22:02.930912+0000","last_peered":"2024-03-27T06:22:02.930912+0000","last_clean":"2024-03-27T06:22:02.930912+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:22:02.930912+0000","last_undegraded":"2024-03-27T06:22:02.930912+0000","last_fullsized":"2024-03-27T06:22:02.930912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:12:12.302147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071357900000000004,"stat_sum":{"num_bytes":5850441,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":350,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"44'320","reported_seq":2425,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.880517+0000","last_change":"2024-03-27T06:18:40.562623+0000","last_active":"2024-03-27T06:22:02.880517+0000","last_peered":"2024-03-27T06:22:02.880517+0000","last_clean":"2024-03-27T06:22:02.880517+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:22:02.880517+0000","last_undegraded":"2024-03-27T06:22:02.880517+0000","last_fullsized":"2024-03-27T06:22:02.880517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:33:31.576614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002466584,"stat_sum":{"num_bytes":14040535,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":20,"num_read_kb":3571,"num_write":321,"num_write_kb":13825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"44'351","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.806369+0000","last_change":"2024-03-27T06:18:40.838197+0000","last_active":"2024-03-27T06:22:02.806369+0000","last_peered":"2024-03-27T06:22:02.806369+0000","last_clean":"2024-03-27T06:22:02.806369+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:22:02.806369+0000","last_undegraded":"2024-03-27T06:22:02.806369+0000","last_fullsized":"2024-03-27T06:22:02.806369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:12:28.032906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056884500000000003,"stat_sum":{"num_bytes":24259091,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":18,"num_read_kb":4113,"num_write":354,"num_write_kb":23807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"44'323","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.745845+0000","last_change":"2024-03-27T06:18:40.830353+0000","last_active":"2024-03-27T06:22:02.745845+0000","last_peered":"2024-03-27T06:22:02.745845+0000","last_clean":"2024-03-27T06:22:02.745845+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:22:02.745845+0000","last_undegraded":"2024-03-27T06:22:02.745845+0000","last_fullsized":"2024-03-27T06:22:02.745845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033930689999999999,"stat_sum":{"num_bytes":15911530,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":12,"num_read_kb":4107,"num_write":326,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"44'320","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.877773+0000","last_change":"2024-03-27T06:18:40.834747+0000","last_active":"2024-03-27T06:22:02.877773+0000","last_peered":"2024-03-27T06:22:02.877773+0000","last_clean":"2024-03-27T06:22:02.877773+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:22:02.877773+0000","last_undegraded":"2024-03-27T06:22:02.877773+0000","last_fullsized":"2024-03-27T06:22:02.877773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:36:36.330080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054729009999999996,"stat_sum":{"num_bytes":6593503,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":20,"num_read_kb":4119,"num_write":318,"num_write_kb":10665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"44'308","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.744529+0000","last_change":"2024-03-27T06:18:40.841044+0000","last_active":"2024-03-27T06:22:02.744529+0000","last_peered":"2024-03-27T06:22:02.744529+0000","last_clean":"2024-03-27T06:22:02.744529+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:22:02.744529+0000","last_undegraded":"2024-03-27T06:22:02.744529+0000","last_fullsized":"2024-03-27T06:22:02.744529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:49.951013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068694350000000001,"stat_sum":{"num_bytes":18722866,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":20,"num_read_kb":9776,"num_write":307,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"44'298","reported_seq":2404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.851195+0000","last_change":"2024-03-27T06:18:40.840622+0000","last_active":"2024-03-27T06:22:02.851195+0000","last_peered":"2024-03-27T06:22:02.851195+0000","last_clean":"2024-03-27T06:22:02.851195+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:22:02.851195+0000","last_undegraded":"2024-03-27T06:22:02.851195+0000","last_fullsized":"2024-03-27T06:22:02.851195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:17:59.498179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070900370000000004,"stat_sum":{"num_bytes":18171100,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":21,"num_read_kb":8210,"num_write":297,"num_write_kb":17901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"44'297","reported_seq":2391,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.000855+0000","last_change":"2024-03-27T06:18:40.840141+0000","last_active":"2024-03-27T06:22:03.000855+0000","last_peered":"2024-03-27T06:22:03.000855+0000","last_clean":"2024-03-27T06:22:03.000855+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:22:03.000855+0000","last_undegraded":"2024-03-27T06:22:03.000855+0000","last_fullsized":"2024-03-27T06:22:03.000855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:41:13.001519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072900689999999997,"stat_sum":{"num_bytes":8281717,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":296,"num_write_kb":12341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"44'291","reported_seq":2396,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.646796+0000","last_change":"2024-03-27T06:18:40.826940+0000","last_active":"2024-03-27T06:22:02.646796+0000","last_peered":"2024-03-27T06:22:02.646796+0000","last_clean":"2024-03-27T06:22:02.646796+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:22:02.646796+0000","last_undegraded":"2024-03-27T06:22:02.646796+0000","last_fullsized":"2024-03-27T06:22:02.646796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029554939999999999,"stat_sum":{"num_bytes":19210972,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":21,"num_read_kb":4116,"num_write":287,"num_write_kb":27094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"44'335","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.553745+0000","last_change":"2024-03-27T06:18:40.827478+0000","last_active":"2024-03-27T06:22:02.553745+0000","last_peered":"2024-03-27T06:22:02.553745+0000","last_clean":"2024-03-27T06:22:02.553745+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:22:02.553745+0000","last_undegraded":"2024-03-27T06:22:02.553745+0000","last_fullsized":"2024-03-27T06:22:02.553745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:34:11.809253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003176613,"stat_sum":{"num_bytes":7395932,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":20,"num_read_kb":4114,"num_write":334,"num_write_kb":11493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.718682+0000","last_change":"2024-03-27T06:18:40.829238+0000","last_active":"2024-03-27T06:22:02.718682+0000","last_peered":"2024-03-27T06:22:02.718682+0000","last_clean":"2024-03-27T06:22:02.718682+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:22:02.718682+0000","last_undegraded":"2024-03-27T06:22:02.718682+0000","last_fullsized":"2024-03-27T06:22:02.718682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:21:38.019283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036973140000000002,"stat_sum":{"num_bytes":23111157,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":349,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.908232+0000","last_change":"2024-03-27T06:18:40.826025+0000","last_active":"2024-03-27T06:22:02.908232+0000","last_peered":"2024-03-27T06:22:02.908232+0000","last_clean":"2024-03-27T06:22:02.908232+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:22:02.908232+0000","last_undegraded":"2024-03-27T06:22:02.908232+0000","last_fullsized":"2024-03-27T06:22:02.908232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001289739,"stat_sum":{"num_bytes":12029261,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":347,"num_write_kb":15208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"44'339","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980167+0000","last_change":"2024-03-27T06:18:40.835662+0000","last_active":"2024-03-27T06:22:02.980167+0000","last_peered":"2024-03-27T06:22:02.980167+0000","last_clean":"2024-03-27T06:22:02.980167+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:22:02.980167+0000","last_undegraded":"2024-03-27T06:22:02.980167+0000","last_fullsized":"2024-03-27T06:22:02.980167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089932469999999994,"stat_sum":{"num_bytes":10346712,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":335,"num_write_kb":14376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"44'303","reported_seq":2414,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.702615+0000","last_change":"2024-03-27T06:18:40.826258+0000","last_active":"2024-03-27T06:22:02.702615+0000","last_peered":"2024-03-27T06:22:02.702615+0000","last_clean":"2024-03-27T06:22:02.702615+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:22:02.702615+0000","last_undegraded":"2024-03-27T06:22:02.702615+0000","last_fullsized":"2024-03-27T06:22:02.702615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:50:38.551582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010652260000000001,"stat_sum":{"num_bytes":8103937,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":302,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"44'319","reported_seq":2426,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.942674+0000","last_change":"2024-03-27T06:18:40.836497+0000","last_active":"2024-03-27T06:22:02.942674+0000","last_peered":"2024-03-27T06:22:02.942674+0000","last_clean":"2024-03-27T06:22:02.942674+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:22:02.942674+0000","last_undegraded":"2024-03-27T06:22:02.942674+0000","last_fullsized":"2024-03-27T06:22:02.942674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:43:03.085499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075917839999999999,"stat_sum":{"num_bytes":14133748,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":23,"num_read_kb":12737,"num_write":315,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"44'345","reported_seq":2453,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.001320+0000","last_change":"2024-03-27T06:18:40.840351+0000","last_active":"2024-03-27T06:22:03.001320+0000","last_peered":"2024-03-27T06:22:03.001320+0000","last_clean":"2024-03-27T06:22:03.001320+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:22:03.001320+0000","last_undegraded":"2024-03-27T06:22:03.001320+0000","last_fullsized":"2024-03-27T06:22:03.001320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:15:43.218527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072459589999999997,"stat_sum":{"num_bytes":23160385,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":24,"num_read_kb":12308,"num_write":342,"num_write_kb":30986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"44'335","reported_seq":2446,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.716055+0000","last_change":"2024-03-27T06:18:40.562416+0000","last_active":"2024-03-27T06:22:02.716055+0000","last_peered":"2024-03-27T06:22:02.716055+0000","last_clean":"2024-03-27T06:22:02.716055+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:22:02.716055+0000","last_undegraded":"2024-03-27T06:22:02.716055+0000","last_fullsized":"2024-03-27T06:22:02.716055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001666102,"stat_sum":{"num_bytes":14662751,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":26,"num_read_kb":4119,"num_write":336,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"44'327","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703008+0000","last_change":"2024-03-27T06:18:40.829281+0000","last_active":"2024-03-27T06:22:02.703008+0000","last_peered":"2024-03-27T06:22:02.703008+0000","last_clean":"2024-03-27T06:22:02.703008+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:22:02.703008+0000","last_undegraded":"2024-03-27T06:22:02.703008+0000","last_fullsized":"2024-03-27T06:22:02.703008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:26:03.545200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024688459999999998,"stat_sum":{"num_bytes":9427787,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":13475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"44'329","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.787322+0000","last_change":"2024-03-27T06:18:40.829763+0000","last_active":"2024-03-27T06:22:02.787322+0000","last_peered":"2024-03-27T06:22:02.787322+0000","last_clean":"2024-03-27T06:22:02.787322+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:22:02.787322+0000","last_undegraded":"2024-03-27T06:22:02.787322+0000","last_fullsized":"2024-03-27T06:22:02.787322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039259519999999999,"stat_sum":{"num_bytes":16419938,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":29,"num_read_kb":18059,"num_write":326,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"44'342","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.905364+0000","last_change":"2024-03-27T06:18:40.826230+0000","last_active":"2024-03-27T06:22:02.905364+0000","last_peered":"2024-03-27T06:22:02.905364+0000","last_clean":"2024-03-27T06:22:02.905364+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:22:02.905364+0000","last_undegraded":"2024-03-27T06:22:02.905364+0000","last_fullsized":"2024-03-27T06:22:02.905364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470573,"stat_sum":{"num_bytes":19403206,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":18,"num_read_kb":8208,"num_write":342,"num_write_kb":19130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"44'316","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730340+0000","last_change":"2024-03-27T06:18:40.829685+0000","last_active":"2024-03-27T06:22:02.730340+0000","last_peered":"2024-03-27T06:22:02.730340+0000","last_clean":"2024-03-27T06:22:02.730340+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:22:02.730340+0000","last_undegraded":"2024-03-27T06:22:02.730340+0000","last_fullsized":"2024-03-27T06:22:02.730340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035867099999999999,"stat_sum":{"num_bytes":14543064,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":314,"num_write_kb":14366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"44'367","reported_seq":2471,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980468+0000","last_change":"2024-03-27T06:18:40.562911+0000","last_active":"2024-03-27T06:22:02.980468+0000","last_peered":"2024-03-27T06:22:02.980468+0000","last_clean":"2024-03-27T06:22:02.980468+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:22:02.980468+0000","last_undegraded":"2024-03-27T06:22:02.980468+0000","last_fullsized":"2024-03-27T06:22:02.980468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:49:45.282319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002151956,"stat_sum":{"num_bytes":5854740,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":367,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"44'308","reported_seq":2411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.589889+0000","last_change":"2024-03-27T06:18:40.827317+0000","last_active":"2024-03-27T06:22:02.589889+0000","last_peered":"2024-03-27T06:22:02.589889+0000","last_clean":"2024-03-27T06:22:02.589889+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:22:02.589889+0000","last_undegraded":"2024-03-27T06:22:02.589889+0000","last_fullsized":"2024-03-27T06:22:02.589889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:28:40.555644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032490380000000001,"stat_sum":{"num_bytes":14027540,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":18,"num_read_kb":8206,"num_write":307,"num_write_kb":13857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"44'310","reported_seq":2410,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.743178+0000","last_change":"2024-03-27T06:18:40.827238+0000","last_active":"2024-03-27T06:22:02.743178+0000","last_peered":"2024-03-27T06:22:02.743178+0000","last_clean":"2024-03-27T06:22:02.743178+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:22:02.743178+0000","last_undegraded":"2024-03-27T06:22:02.743178+0000","last_fullsized":"2024-03-27T06:22:02.743178+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:35:39.617870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003068571,"stat_sum":{"num_bytes":18001138,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":16,"num_read_kb":7383,"num_write":307,"num_write_kb":21016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"44'351","reported_seq":2458,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.617101+0000","last_change":"2024-03-27T06:18:40.829449+0000","last_active":"2024-03-27T06:22:02.617101+0000","last_peered":"2024-03-27T06:22:02.617101+0000","last_clean":"2024-03-27T06:22:02.617101+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:22:02.617101+0000","last_undegraded":"2024-03-27T06:22:02.617101+0000","last_fullsized":"2024-03-27T06:22:02.617101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:47:20.901338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036751610000000001,"stat_sum":{"num_bytes":22534890,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":22,"num_read_kb":12306,"num_write":351,"num_write_kb":22191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"44'351","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.911800+0000","last_change":"2024-03-27T06:18:40.836714+0000","last_active":"2024-03-27T06:22:02.911800+0000","last_peered":"2024-03-27T06:22:02.911800+0000","last_clean":"2024-03-27T06:22:02.911800+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:22:02.911800+0000","last_undegraded":"2024-03-27T06:22:02.911800+0000","last_fullsized":"2024-03-27T06:22:02.911800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044159999999999,"stat_sum":{"num_bytes":14250100,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":24,"num_read_kb":7593,"num_write":351,"num_write_kb":14104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"44'375","reported_seq":2479,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.663268+0000","last_change":"2024-03-27T06:18:40.825940+0000","last_active":"2024-03-27T06:22:02.663268+0000","last_peered":"2024-03-27T06:22:02.663268+0000","last_clean":"2024-03-27T06:22:02.663268+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:22:02.663268+0000","last_undegraded":"2024-03-27T06:22:02.663268+0000","last_fullsized":"2024-03-27T06:22:02.663268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:54:24.095203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011795639999999999,"stat_sum":{"num_bytes":10303903,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":19,"num_read_kb":1482,"num_write":377,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"44'371","reported_seq":2484,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.738005+0000","last_change":"2024-03-27T06:18:40.561817+0000","last_active":"2024-03-27T06:22:02.738005+0000","last_peered":"2024-03-27T06:22:02.738005+0000","last_clean":"2024-03-27T06:22:02.738005+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:22:02.738005+0000","last_undegraded":"2024-03-27T06:22:02.738005+0000","last_fullsized":"2024-03-27T06:22:02.738005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:57:04.359511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016911179999999999,"stat_sum":{"num_bytes":11109721,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":28,"num_read_kb":4123,"num_write":370,"num_write_kb":11044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"44'320","reported_seq":2423,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.653651+0000","last_change":"2024-03-27T06:18:40.831829+0000","last_active":"2024-03-27T06:22:02.653651+0000","last_peered":"2024-03-27T06:22:02.653651+0000","last_clean":"2024-03-27T06:22:02.653651+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:22:02.653651+0000","last_undegraded":"2024-03-27T06:22:02.653651+0000","last_fullsized":"2024-03-27T06:22:02.653651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:18:03.157765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057058689999999997,"stat_sum":{"num_bytes":18559363,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":18,"num_read_kb":4112,"num_write":320,"num_write_kb":18284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"43'17","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283493+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:21:30.283493+0000","last_peered":"2024-03-27T06:21:30.283493+0000","last_clean":"2024-03-27T06:21:30.283493+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:21:30.283493+0000","last_undegraded":"2024-03-27T06:21:30.283493+0000","last_fullsized":"2024-03-27T06:21:30.283493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195296,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":18,"num_write_kb":4248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'311","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.773731+0000","last_change":"2024-03-27T06:18:40.562829+0000","last_active":"2024-03-27T06:22:02.773731+0000","last_peered":"2024-03-27T06:22:02.773731+0000","last_clean":"2024-03-27T06:22:02.773731+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:22:02.773731+0000","last_undegraded":"2024-03-27T06:22:02.773731+0000","last_fullsized":"2024-03-27T06:22:02.773731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:17:17.674549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018454000000000001,"stat_sum":{"num_bytes":6743271,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":310,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"44'253","reported_seq":326,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.246658+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:21:55.246658+0000","last_peered":"2024-03-27T06:21:55.246658+0000","last_clean":"2024-03-27T06:21:55.246658+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:21:55.246658+0000","last_undegraded":"2024-03-27T06:21:55.246658+0000","last_fullsized":"2024-03-27T06:21:55.246658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":8577,"num_write":252,"num_write_kb":12472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"43'62","reported_seq":132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.244913+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:21:55.244913+0000","last_peered":"2024-03-27T06:21:55.244913+0000","last_clean":"2024-03-27T06:21:55.244913+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:21:55.244913+0000","last_undegraded":"2024-03-27T06:21:55.244913+0000","last_fullsized":"2024-03-27T06:21:55.244913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":63,"num_write_kb":4164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"44'399","reported_seq":2511,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.700284+0000","last_change":"2024-03-27T06:18:40.830513+0000","last_active":"2024-03-27T06:22:02.700284+0000","last_peered":"2024-03-27T06:22:02.700284+0000","last_clean":"2024-03-27T06:22:02.700284+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:22:02.700284+0000","last_undegraded":"2024-03-27T06:22:02.700284+0000","last_fullsized":"2024-03-27T06:22:02.700284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003103719,"stat_sum":{"num_bytes":18931276,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":28,"num_read_kb":12930,"num_write":396,"num_write_kb":26887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"44'377","reported_seq":442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:53.574559+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:21:53.574559+0000","last_peered":"2024-03-27T06:21:53.574559+0000","last_clean":"2024-03-27T06:21:53.574559+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:21:53.574559+0000","last_undegraded":"2024-03-27T06:21:53.574559+0000","last_fullsized":"2024-03-27T06:21:53.574559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":374,"num_write_kb":12507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"44'335","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.674076+0000","last_change":"2024-03-27T06:18:40.562924+0000","last_active":"2024-03-27T06:22:02.674076+0000","last_peered":"2024-03-27T06:22:02.674076+0000","last_clean":"2024-03-27T06:22:02.674076+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:22:02.674076+0000","last_undegraded":"2024-03-27T06:22:02.674076+0000","last_fullsized":"2024-03-27T06:22:02.674076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:17:24.783114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019163050000000001,"stat_sum":{"num_bytes":19664158,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":13845,"num_write":329,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"43'10","reported_seq":76,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.401768+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:21:30.401768+0000","last_peered":"2024-03-27T06:21:30.401768+0000","last_clean":"2024-03-27T06:21:30.401768+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:21:30.401768+0000","last_undegraded":"2024-03-27T06:21:30.401768+0000","last_fullsized":"2024-03-27T06:21:30.401768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"44'351","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.986843+0000","last_change":"2024-03-27T06:18:40.563693+0000","last_active":"2024-03-27T06:22:02.986843+0000","last_peered":"2024-03-27T06:22:02.986843+0000","last_clean":"2024-03-27T06:22:02.986843+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:22:02.986843+0000","last_undegraded":"2024-03-27T06:22:02.986843+0000","last_fullsized":"2024-03-27T06:22:02.986843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026075790000000001,"stat_sum":{"num_bytes":22946854,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":25,"num_read_kb":12308,"num_write":351,"num_write_kb":22593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"44'107","reported_seq":173,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.610595+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:21:29.610595+0000","last_peered":"2024-03-27T06:21:29.610595+0000","last_clean":"2024-03-27T06:21:29.610595+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:21:29.610595+0000","last_undegraded":"2024-03-27T06:21:29.610595+0000","last_fullsized":"2024-03-27T06:21:29.610595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":106,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"44'320","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.808658+0000","last_change":"2024-03-27T06:18:40.830168+0000","last_active":"2024-03-27T06:22:02.808658+0000","last_peered":"2024-03-27T06:22:02.808658+0000","last_clean":"2024-03-27T06:22:02.808658+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:22:02.808658+0000","last_undegraded":"2024-03-27T06:22:02.808658+0000","last_fullsized":"2024-03-27T06:22:02.808658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003650304,"stat_sum":{"num_bytes":10082281,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":22,"num_read_kb":4115,"num_write":319,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"43'280","reported_seq":346,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.210794+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:21:29.210794+0000","last_peered":"2024-03-27T06:21:29.210794+0000","last_clean":"2024-03-27T06:21:29.210794+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:21:29.210794+0000","last_undegraded":"2024-03-27T06:21:29.210794+0000","last_fullsized":"2024-03-27T06:21:29.210794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":280,"num_write_kb":4300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'315","reported_seq":2416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.750367+0000","last_change":"2024-03-27T06:18:40.835973+0000","last_active":"2024-03-27T06:22:02.750367+0000","last_peered":"2024-03-27T06:22:02.750367+0000","last_clean":"2024-03-27T06:22:02.750367+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:22:02.750367+0000","last_undegraded":"2024-03-27T06:22:02.750367+0000","last_fullsized":"2024-03-27T06:22:02.750367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084315699999999993,"stat_sum":{"num_bytes":18808790,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":315,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"44'127","reported_seq":197,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:48.532269+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:21:48.532269+0000","last_peered":"2024-03-27T06:21:48.532269+0000","last_clean":"2024-03-27T06:21:48.532269+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:21:48.532269+0000","last_undegraded":"2024-03-27T06:21:48.532269+0000","last_fullsized":"2024-03-27T06:21:48.532269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":8203,"num_write":123,"num_write_kb":12364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"44'327","reported_seq":2430,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.742161+0000","last_change":"2024-03-27T06:18:40.841177+0000","last_active":"2024-03-27T06:22:02.742161+0000","last_peered":"2024-03-27T06:22:02.742161+0000","last_clean":"2024-03-27T06:22:02.742161+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:22:02.742161+0000","last_undegraded":"2024-03-27T06:22:02.742161+0000","last_fullsized":"2024-03-27T06:22:02.742161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:44:07.500058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050386370000000003,"stat_sum":{"num_bytes":2468399,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":327,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"44'144","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.248730+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:21:55.248730+0000","last_peered":"2024-03-27T06:21:55.248730+0000","last_clean":"2024-03-27T06:21:55.248730+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:21:55.248730+0000","last_undegraded":"2024-03-27T06:21:55.248730+0000","last_fullsized":"2024-03-27T06:21:55.248730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":8224,"num_write":145,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'341","reported_seq":2450,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.612638+0000","last_change":"2024-03-27T06:18:40.826892+0000","last_active":"2024-03-27T06:22:02.612638+0000","last_peered":"2024-03-27T06:22:02.612638+0000","last_clean":"2024-03-27T06:22:02.612638+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:22:02.612638+0000","last_undegraded":"2024-03-27T06:22:02.612638+0000","last_fullsized":"2024-03-27T06:22:02.612638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002905866,"stat_sum":{"num_bytes":19666163,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":25,"num_read_kb":16403,"num_write":336,"num_write_kb":27577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"44'6","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.214987+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:21:29.214987+0000","last_peered":"2024-03-27T06:21:29.214987+0000","last_clean":"2024-03-27T06:21:29.214987+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:21:29.214987+0000","last_undegraded":"2024-03-27T06:21:29.214987+0000","last_fullsized":"2024-03-27T06:21:29.214987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"44'346","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.820209+0000","last_change":"2024-03-27T06:18:40.837185+0000","last_active":"2024-03-27T06:22:02.820209+0000","last_peered":"2024-03-27T06:22:02.820209+0000","last_clean":"2024-03-27T06:22:02.820209+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:22:02.820209+0000","last_undegraded":"2024-03-27T06:22:02.820209+0000","last_fullsized":"2024-03-27T06:22:02.820209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010165321999999999,"stat_sum":{"num_bytes":10617237,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":345,"num_write_kb":10547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"44'12","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981540+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:22:03.981540+0000","last_peered":"2024-03-27T06:22:03.981540+0000","last_clean":"2024-03-27T06:22:03.981540+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:22:03.981540+0000","last_undegraded":"2024-03-27T06:22:03.981540+0000","last_fullsized":"2024-03-27T06:22:03.981540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":11,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"44'347","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.588028+0000","last_change":"2024-03-27T06:18:40.831916+0000","last_active":"2024-03-27T06:22:02.588028+0000","last_peered":"2024-03-27T06:22:02.588028+0000","last_clean":"2024-03-27T06:22:02.588028+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:22:02.588028+0000","last_undegraded":"2024-03-27T06:22:02.588028+0000","last_fullsized":"2024-03-27T06:22:02.588028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:33:54.035813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054212890000000001,"stat_sum":{"num_bytes":24812922,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":17,"num_read_kb":8206,"num_write":342,"num_write_kb":26402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"44'278","reported_seq":359,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250589+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:21:55.250589+0000","last_peered":"2024-03-27T06:21:55.250589+0000","last_clean":"2024-03-27T06:21:55.250589+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:21:55.250589+0000","last_undegraded":"2024-03-27T06:21:55.250589+0000","last_fullsized":"2024-03-27T06:21:55.250589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":283,"num_write_kb":8391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"44'358","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.826841+0000","last_change":"2024-03-27T06:18:40.834701+0000","last_active":"2024-03-27T06:22:02.826841+0000","last_peered":"2024-03-27T06:22:02.826841+0000","last_clean":"2024-03-27T06:22:02.826841+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:22:02.826841+0000","last_undegraded":"2024-03-27T06:22:02.826841+0000","last_fullsized":"2024-03-27T06:22:02.826841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083622499999999999,"stat_sum":{"num_bytes":12063538,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":20,"num_read_kb":12305,"num_write":353,"num_write_kb":20111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"44'29","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212511+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:21:34.212511+0000","last_peered":"2024-03-27T06:21:34.212511+0000","last_clean":"2024-03-27T06:21:34.212511+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:21:34.212511+0000","last_undegraded":"2024-03-27T06:21:34.212511+0000","last_fullsized":"2024-03-27T06:21:34.212511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4103,"num_write":34,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'291","reported_seq":2389,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.989310+0000","last_change":"2024-03-27T06:18:40.563367+0000","last_active":"2024-03-27T06:22:02.989310+0000","last_peered":"2024-03-27T06:22:02.989310+0000","last_clean":"2024-03-27T06:22:02.989310+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:22:02.989310+0000","last_undegraded":"2024-03-27T06:22:02.989310+0000","last_fullsized":"2024-03-27T06:22:02.989310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022904700000000002,"stat_sum":{"num_bytes":11066744,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":12134,"num_write":288,"num_write_kb":15054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"44'28","reported_seq":93,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601016+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:21:29.601016+0000","last_peered":"2024-03-27T06:21:29.601016+0000","last_clean":"2024-03-27T06:21:29.601016+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:21:29.601016+0000","last_undegraded":"2024-03-27T06:21:29.601016+0000","last_fullsized":"2024-03-27T06:21:29.601016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":30,"num_write_kb":4134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"44'327","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.837377+0000","last_change":"2024-03-27T06:18:40.841709+0000","last_active":"2024-03-27T06:22:02.837377+0000","last_peered":"2024-03-27T06:22:02.837377+0000","last_clean":"2024-03-27T06:22:02.837377+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:22:02.837377+0000","last_undegraded":"2024-03-27T06:22:02.837377+0000","last_fullsized":"2024-03-27T06:22:02.837377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:43:54.568781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050322370000000002,"stat_sum":{"num_bytes":14587209,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":24,"num_read_kb":12307,"num_write":322,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"44'23","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283694+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:21:30.283694+0000","last_peered":"2024-03-27T06:21:30.283694+0000","last_clean":"2024-03-27T06:21:30.283694+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:21:30.283694+0000","last_undegraded":"2024-03-27T06:21:30.283694+0000","last_fullsized":"2024-03-27T06:21:30.283694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":23,"num_write_kb":4120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"44'356","reported_seq":2457,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.998863+0000","last_change":"2024-03-27T06:18:40.840461+0000","last_active":"2024-03-27T06:22:02.998863+0000","last_peered":"2024-03-27T06:22:02.998863+0000","last_clean":"2024-03-27T06:22:02.998863+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:22:02.998863+0000","last_undegraded":"2024-03-27T06:22:02.998863+0000","last_fullsized":"2024-03-27T06:22:02.998863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:35.295705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070052669999999999,"stat_sum":{"num_bytes":11205183,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":352,"num_write_kb":11129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"44'29","reported_seq":101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:58.564457+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:21:58.564457+0000","last_peered":"2024-03-27T06:21:58.564457+0000","last_clean":"2024-03-27T06:21:58.564457+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:21:58.564457+0000","last_undegraded":"2024-03-27T06:21:58.564457+0000","last_fullsized":"2024-03-27T06:21:58.564457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":32,"num_write_kb":4138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"44'323","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.968996+0000","last_change":"2024-03-27T06:18:40.835886+0000","last_active":"2024-03-27T06:22:02.968996+0000","last_peered":"2024-03-27T06:22:02.968996+0000","last_clean":"2024-03-27T06:22:02.968996+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:22:02.968996+0000","last_undegraded":"2024-03-27T06:22:02.968996+0000","last_fullsized":"2024-03-27T06:22:02.968996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:18:41.783945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086058899999999997,"stat_sum":{"num_bytes":14608246,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":19,"num_read_kb":4113,"num_write":322,"num_write_kb":18530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.384769+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:21:30.384769+0000","last_peered":"2024-03-27T06:21:30.384769+0000","last_clean":"2024-03-27T06:21:30.384769+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:21:30.384769+0000","last_undegraded":"2024-03-27T06:21:30.384769+0000","last_fullsized":"2024-03-27T06:21:30.384769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'84","reported_seq":152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212620+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:21:34.212620+0000","last_peered":"2024-03-27T06:21:34.212620+0000","last_clean":"2024-03-27T06:21:34.212620+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:21:34.212620+0000","last_undegraded":"2024-03-27T06:21:34.212620+0000","last_fullsized":"2024-03-27T06:21:34.212620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":86,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'321","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.778867+0000","last_change":"2024-03-27T06:18:40.563107+0000","last_active":"2024-03-27T06:22:02.778867+0000","last_peered":"2024-03-27T06:22:02.778867+0000","last_clean":"2024-03-27T06:22:02.778867+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:22:02.778867+0000","last_undegraded":"2024-03-27T06:22:02.778867+0000","last_fullsized":"2024-03-27T06:22:02.778867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:37:09.221434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002005691,"stat_sum":{"num_bytes":10118185,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":14,"num_read_kb":4109,"num_write":319,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"44'228","reported_seq":300,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.247793+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:21:55.247793+0000","last_peered":"2024-03-27T06:21:55.247793+0000","last_clean":"2024-03-27T06:21:55.247793+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:21:55.247793+0000","last_undegraded":"2024-03-27T06:21:55.247793+0000","last_fullsized":"2024-03-27T06:21:55.247793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":8210,"num_write":225,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'326","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.648394+0000","last_change":"2024-03-27T06:18:40.825563+0000","last_active":"2024-03-27T06:22:02.648394+0000","last_peered":"2024-03-27T06:22:02.648394+0000","last_clean":"2024-03-27T06:22:02.648394+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:22:02.648394+0000","last_undegraded":"2024-03-27T06:22:02.648394+0000","last_fullsized":"2024-03-27T06:22:02.648394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001779484,"stat_sum":{"num_bytes":10043455,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":324,"num_write_kb":9981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"44'90","reported_seq":241,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.244489+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:21:55.244489+0000","last_peered":"2024-03-27T06:21:55.244489+0000","last_clean":"2024-03-27T06:21:55.244489+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:21:55.244489+0000","last_undegraded":"2024-03-27T06:21:55.244489+0000","last_fullsized":"2024-03-27T06:21:55.244489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534287,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":595,"num_write":89,"num_write_kb":601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"44'332","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703670+0000","last_change":"2024-03-27T06:18:40.830316+0000","last_active":"2024-03-27T06:22:02.703670+0000","last_peered":"2024-03-27T06:22:02.703670+0000","last_clean":"2024-03-27T06:22:02.703670+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:22:02.703670+0000","last_undegraded":"2024-03-27T06:22:02.703670+0000","last_fullsized":"2024-03-27T06:22:02.703670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031179929999999999,"stat_sum":{"num_bytes":2779321,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":332,"num_write_kb":2894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"44'41","reported_seq":106,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.216718+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:21:34.216718+0000","last_peered":"2024-03-27T06:21:34.216718+0000","last_clean":"2024-03-27T06:21:34.216718+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:21:34.216718+0000","last_undegraded":"2024-03-27T06:21:34.216718+0000","last_fullsized":"2024-03-27T06:21:34.216718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":36,"num_write_kb":8227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"44'346","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.723176+0000","last_change":"2024-03-27T06:18:40.826852+0000","last_active":"2024-03-27T06:22:02.723176+0000","last_peered":"2024-03-27T06:22:02.723176+0000","last_clean":"2024-03-27T06:22:02.723176+0000","last_became_active":"2024-03-27T06:14:49.853879+0000","last_became_peered":"2024-03-27T06:14:49.853879+0000","last_unstale":"2024-03-27T06:22:02.723176+0000","last_undegraded":"2024-03-27T06:22:02.723176+0000","last_fullsized":"2024-03-27T06:22:02.723176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:54:30.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022092909999999999,"stat_sum":{"num_bytes":5308376,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":346,"num_write_kb":5365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"44'78","reported_seq":141,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:37.150979+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:21:37.150979+0000","last_peered":"2024-03-27T06:21:37.150979+0000","last_clean":"2024-03-27T06:21:37.150979+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:21:37.150979+0000","last_undegraded":"2024-03-27T06:21:37.150979+0000","last_fullsized":"2024-03-27T06:21:37.150979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":4150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"44'340","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.631346+0000","last_change":"2024-03-27T06:18:40.563208+0000","last_active":"2024-03-27T06:22:02.631346+0000","last_peered":"2024-03-27T06:22:02.631346+0000","last_clean":"2024-03-27T06:22:02.631346+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:22:02.631346+0000","last_undegraded":"2024-03-27T06:22:02.631346+0000","last_fullsized":"2024-03-27T06:22:02.631346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002460744,"stat_sum":{"num_bytes":18473602,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":23,"num_read_kb":12305,"num_write":340,"num_write_kb":22314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'587","reported_seq":652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:40.150130+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:21:40.150130+0000","last_peered":"2024-03-27T06:21:40.150130+0000","last_clean":"2024-03-27T06:21:40.150130+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:21:40.150130+0000","last_undegraded":"2024-03-27T06:21:40.150130+0000","last_fullsized":"2024-03-27T06:21:40.150130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":587,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"44'331","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.580275+0000","last_change":"2024-03-27T06:18:40.827659+0000","last_active":"2024-03-27T06:22:02.580275+0000","last_peered":"2024-03-27T06:22:02.580275+0000","last_clean":"2024-03-27T06:22:02.580275+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:22:02.580275+0000","last_undegraded":"2024-03-27T06:22:02.580275+0000","last_fullsized":"2024-03-27T06:22:02.580275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032966190000000002,"stat_sum":{"num_bytes":6064397,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":331,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"44'137","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.629820+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:22:03.629820+0000","last_peered":"2024-03-27T06:22:03.629820+0000","last_clean":"2024-03-27T06:22:03.629820+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:22:03.629820+0000","last_undegraded":"2024-03-27T06:22:03.629820+0000","last_fullsized":"2024-03-27T06:22:03.629820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":322,"num_write_kb":2012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"44'317","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.566427+0000","last_change":"2024-03-27T06:18:40.562700+0000","last_active":"2024-03-27T06:22:02.566427+0000","last_peered":"2024-03-27T06:22:02.566427+0000","last_clean":"2024-03-27T06:22:02.566427+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:22:02.566427+0000","last_undegraded":"2024-03-27T06:22:02.566427+0000","last_fullsized":"2024-03-27T06:22:02.566427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:57:15.843788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017608820000000001,"stat_sum":{"num_bytes":20225923,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":318,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"43'79","reported_seq":145,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.215719+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:21:29.215719+0000","last_peered":"2024-03-27T06:21:29.215719+0000","last_clean":"2024-03-27T06:21:29.215719+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:21:29.215719+0000","last_undegraded":"2024-03-27T06:21:29.215719+0000","last_fullsized":"2024-03-27T06:21:29.215719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":78,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'350","reported_seq":2469,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.556934+0000","last_change":"2024-03-27T06:18:40.562997+0000","last_active":"2024-03-27T06:22:02.556934+0000","last_peered":"2024-03-27T06:22:02.556934+0000","last_clean":"2024-03-27T06:22:02.556934+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:22:02.556934+0000","last_undegraded":"2024-03-27T06:22:02.556934+0000","last_fullsized":"2024-03-27T06:22:02.556934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:19:49.684960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018869889999999999,"stat_sum":{"num_bytes":32233430,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":35,"num_read_kb":18086,"num_write":345,"num_write_kb":31650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"44'31","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.609369+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:21:29.609369+0000","last_peered":"2024-03-27T06:21:29.609369+0000","last_clean":"2024-03-27T06:21:29.609369+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:21:29.609369+0000","last_undegraded":"2024-03-27T06:21:29.609369+0000","last_fullsized":"2024-03-27T06:21:29.609369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":35,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'347","reported_seq":2451,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.890096+0000","last_change":"2024-03-27T06:18:40.836137+0000","last_active":"2024-03-27T06:22:02.890096+0000","last_peered":"2024-03-27T06:22:02.890096+0000","last_clean":"2024-03-27T06:22:02.890096+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:22:02.890096+0000","last_undegraded":"2024-03-27T06:22:02.890096+0000","last_fullsized":"2024-03-27T06:22:02.890096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081323179999999995,"stat_sum":{"num_bytes":22949808,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":346,"num_write_kb":22595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"44'118","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601177+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:21:29.601177+0000","last_peered":"2024-03-27T06:21:29.601177+0000","last_clean":"2024-03-27T06:21:29.601177+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:21:29.601177+0000","last_undegraded":"2024-03-27T06:21:29.601177+0000","last_fullsized":"2024-03-27T06:21:29.601177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":120,"num_write_kb":4433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"44'355","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.550157+0000","last_change":"2024-03-27T06:18:40.562590+0000","last_active":"2024-03-27T06:22:02.550157+0000","last_peered":"2024-03-27T06:22:02.550157+0000","last_clean":"2024-03-27T06:22:02.550157+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:22:02.550157+0000","last_undegraded":"2024-03-27T06:22:02.550157+0000","last_fullsized":"2024-03-27T06:22:02.550157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:13:24.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001751964,"stat_sum":{"num_bytes":19413938,"num_objects":185,"num_object_clones":0,"num_object_copies":370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":21,"num_read_kb":12304,"num_write":357,"num_write_kb":23191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"44'26","reported_seq":91,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:33.720499+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:21:33.720499+0000","last_peered":"2024-03-27T06:21:33.720499+0000","last_clean":"2024-03-27T06:21:33.720499+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:21:33.720499+0000","last_undegraded":"2024-03-27T06:21:33.720499+0000","last_fullsized":"2024-03-27T06:21:33.720499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":4098,"num_write":30,"num_write_kb":4123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'319","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.572064+0000","last_change":"2024-03-27T06:18:40.826508+0000","last_active":"2024-03-27T06:22:02.572064+0000","last_peered":"2024-03-27T06:22:02.572064+0000","last_clean":"2024-03-27T06:22:02.572064+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:22:02.572064+0000","last_undegraded":"2024-03-27T06:22:02.572064+0000","last_fullsized":"2024-03-27T06:22:02.572064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:40.745990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001518585,"stat_sum":{"num_bytes":15553910,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":17,"num_read_kb":5089,"num_write":319,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"44'200","reported_seq":268,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981086+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:22:03.981086+0000","last_peered":"2024-03-27T06:22:03.981086+0000","last_clean":"2024-03-27T06:22:03.981086+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:22:03.981086+0000","last_undegraded":"2024-03-27T06:22:03.981086+0000","last_fullsized":"2024-03-27T06:22:03.981086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9715716,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":197,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'347","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.917806+0000","last_change":"2024-03-27T06:18:40.836916+0000","last_active":"2024-03-27T06:22:02.917806+0000","last_peered":"2024-03-27T06:22:02.917806+0000","last_clean":"2024-03-27T06:22:02.917806+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:22:02.917806+0000","last_undegraded":"2024-03-27T06:22:02.917806+0000","last_fullsized":"2024-03-27T06:22:02.917806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:26:09.736687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044719999999999,"stat_sum":{"num_bytes":10744528,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":347,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'307","reported_seq":2417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.975860+0000","last_change":"2024-03-27T06:18:40.835719+0000","last_active":"2024-03-27T06:22:02.975860+0000","last_peered":"2024-03-27T06:22:02.975860+0000","last_clean":"2024-03-27T06:22:02.975860+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:22:02.975860+0000","last_undegraded":"2024-03-27T06:22:02.975860+0000","last_fullsized":"2024-03-27T06:22:02.975860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086575760000000002,"stat_sum":{"num_bytes":8008268,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4119,"num_write":307,"num_write_kb":12040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"30'22","reported_seq":87,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.611265+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:21:29.611265+0000","last_peered":"2024-03-27T06:21:29.611265+0000","last_clean":"2024-03-27T06:21:29.611265+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:21:29.611265+0000","last_undegraded":"2024-03-27T06:21:29.611265+0000","last_fullsized":"2024-03-27T06:21:29.611265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":22,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"44'335","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.726938+0000","last_change":"2024-03-27T06:18:40.835158+0000","last_active":"2024-03-27T06:22:02.726938+0000","last_peered":"2024-03-27T06:22:02.726938+0000","last_clean":"2024-03-27T06:22:02.726938+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:22:02.726938+0000","last_undegraded":"2024-03-27T06:22:02.726938+0000","last_fullsized":"2024-03-27T06:22:02.726938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:10:12.399076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061971559999999997,"stat_sum":{"num_bytes":27398907,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":15,"num_read_kb":8205,"num_write":334,"num_write_kb":31029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"44'55","reported_seq":125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.239947+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:21:55.239947+0000","last_peered":"2024-03-27T06:21:55.239947+0000","last_clean":"2024-03-27T06:21:55.239947+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:21:55.239947+0000","last_undegraded":"2024-03-27T06:21:55.239947+0000","last_fullsized":"2024-03-27T06:21:55.239947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":12312,"num_write":46,"num_write_kb":12316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"44'312","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730202+0000","last_change":"2024-03-27T06:18:40.829954+0000","last_active":"2024-03-27T06:22:02.730202+0000","last_peered":"2024-03-27T06:22:02.730202+0000","last_clean":"2024-03-27T06:22:02.730202+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:22:02.730202+0000","last_undegraded":"2024-03-27T06:22:02.730202+0000","last_fullsized":"2024-03-27T06:22:02.730202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036426230000000002,"stat_sum":{"num_bytes":14422279,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":21,"num_read_kb":4923,"num_write":311,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"44'81","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978121+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:22:03.978121+0000","last_peered":"2024-03-27T06:22:03.978121+0000","last_clean":"2024-03-27T06:22:03.978121+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:22:03.978121+0000","last_undegraded":"2024-03-27T06:22:03.978121+0000","last_fullsized":"2024-03-27T06:22:03.978121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":75,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"44'342","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.795505+0000","last_change":"2024-03-27T06:18:40.830116+0000","last_active":"2024-03-27T06:22:02.795505+0000","last_peered":"2024-03-27T06:22:02.795505+0000","last_clean":"2024-03-27T06:22:02.795505+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:22:02.795505+0000","last_undegraded":"2024-03-27T06:22:02.795505+0000","last_fullsized":"2024-03-27T06:22:02.795505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036711629999999999,"stat_sum":{"num_bytes":15186793,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":8204,"num_write":343,"num_write_kb":14950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"44'129","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.980283+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:22:03.980283+0000","last_peered":"2024-03-27T06:22:03.980283+0000","last_clean":"2024-03-27T06:22:03.980283+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:22:03.980283+0000","last_undegraded":"2024-03-27T06:22:03.980283+0000","last_fullsized":"2024-03-27T06:22:03.980283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":4136,"num_write":129,"num_write_kb":8304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940893603,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":1256,"num_read_kb":394723,"num_write":21174,"num_write_kb":1064721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1712472064,"data_stored":1880247110,"data_compressed":164906076,"data_compressed_allocated":182468608,"data_compressed_original":405446656,"omap_allocated":0,"internal_metadata":0},"log_size":21254,"ondisk_log_size":21254,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182221515,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":453,"num_read_kb":120084,"num_write":3948,"num_write_kb":183247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":91,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":80830464,"data_stored":364443030,"data_compressed":49732790,"data_compressed_allocated":62275584,"data_compressed_original":346046464,"omap_allocated":0,"internal_metadata":0},"log_size":3768,"ondisk_log_size":3768,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170540,"kb_used_data":143524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94201300,"statfs":{"total":96636764160,"available":96462131200,"internally_reserved":0,"allocated":146968576,"data_stored":174700599,"data_compressed":11348325,"data_compressed_allocated":13275136,"data_compressed_original":46198784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":213540,"kb_used_data":169884,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94158300,"statfs":{"total":96636764160,"available":96418099200,"internally_reserved":0,"allocated":173961216,"data_stored":215859427,"data_compressed":22390241,"data_compressed_allocated":25247744,"data_compressed_original":73641984,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291908,"kb_used_data":242684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94079932,"statfs":{"total":96636764160,"available":96337850368,"internally_reserved":0,"allocated":248508416,"data_stored":304860446,"data_compressed":28268738,"data_compressed_allocated":32043008,"data_compressed_original":96636928,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266772,"kb_used_data":217548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94105068,"statfs":{"total":96636764160,"available":96363589632,"internally_reserved":0,"allocated":222769152,"data_stored":282203639,"data_compressed":27135224,"data_compressed_allocated":30642176,"data_compressed_original":95821824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313816,"kb_used_data":264592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94058024,"statfs":{"total":96636764160,"available":96315416576,"internally_reserved":0,"allocated":270942208,"data_stored":358995823,"data_compressed":39540899,"data_compressed_allocated":45490176,"data_compressed_original":142065664,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267308,"kb_used_data":222244,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94104532,"statfs":{"total":96636764160,"available":96363040768,"internally_reserved":0,"allocated":227577856,"data_stored":275207385,"data_compressed":26559283,"data_compressed_allocated":30052352,"data_compressed_original":85663744,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246944,"kb_used_data":201880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94124896,"statfs":{"total":96636764160,"available":96383893504,"internally_reserved":0,"allocated":206725120,"data_stored":255586545,"data_compressed":23288424,"data_compressed_allocated":26611712,"data_compressed_original":81715200,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360220,"kb_used_data":292948,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94011620,"statfs":{"total":96636764160,"available":96267898880,"internally_reserved":0,"allocated":299978752,"data_stored":380226852,"data_compressed":36172470,"data_compressed_allocated":41472000,"data_compressed_original":130895872,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":67110666,"data_compressed":8915435,"data_compressed_allocated":11333632,"data_compressed_original":64311296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7794688,"data_stored":37748896,"data_compressed":4858381,"data_compressed_allocated":6283264,"data_compressed_original":36245504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6713344,"data_stored":30687942,"data_compressed":4449598,"data_compressed_allocated":5500928,"data_compressed_original":29491200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19513344,"data_stored":81020596,"data_compressed":11156334,"data_compressed_allocated":14245888,"data_compressed_original":75771904,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":50333888,"data_compressed":7205527,"data_compressed_allocated":8712192,"data_compressed_original":46735360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8495104,"data_stored":42478411,"data_compressed":5618062,"data_compressed_allocated":6979584,"data_compressed_original":40984576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5763072,"data_stored":29361702,"data_compressed":4140996,"data_compressed_allocated":5042176,"data_compressed_original":28651520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":25700929,"data_compressed":3388457,"data_compressed_allocated":4177920,"data_compressed_original":23855104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285327360,"data_stored":312894956,"data_compressed":27257035,"data_compressed_allocated":30138368,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198365184,"data_stored":217026051,"data_compressed":18397674,"data_compressed_allocated":20283392,"data_compressed_original":44896256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220364800,"data_stored":244298213,"data_compressed":22109685,"data_compressed_allocated":24551424,"data_compressed_original":56172544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250929152,"data_stored":277753997,"data_compressed":28384565,"data_compressed_allocated":31244288,"data_compressed_original":66293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209920000,"data_stored":231648521,"data_compressed":19929697,"data_compressed_allocated":21929984,"data_compressed_original":49086464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239448064,"data_stored":261570437,"data_compressed":22618307,"data_compressed_allocated":25018368,"data_compressed_original":55078912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167698432,"data_stored":186276495,"data_compressed":18249245,"data_compressed_allocated":20205568,"data_compressed_original":44990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140419072,"data_stored":148778440,"data_compressed":7959868,"data_compressed_allocated":9097216,"data_compressed_original":22343680,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:22:09.973 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T06:22:09.973 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:22:10.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.226+0000 7fa1cf200640 1 Processor -- start 2024-03-27T06:22:10.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.226+0000 7fa1cf200640 1 -- start start 2024-03-27T06:22:10.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cf200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 0x7fa1c80c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cf200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa1c80c7ee0 0x7fa1c80c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cf200640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c80c6760 con 0x7fa1c81529d0 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cf200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c80c7050 con 0x7fa1c80c3b50 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cf200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c80c7860 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1c7e00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa1c81529d0 0x7fa1c815c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:60456/0 (socket says 172.21.15.132:60456) 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1c7e00640 1 -- 172.21.15.132:0/4179521550 learned_addr learned my addr 172.21.15.132:0/4179521550 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:10.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.228+0000 7fa1cd200640 1 --2- 172.21.15.132:0/4179521550 >> v2:172.21.15.134:3300/0 conn(0x7fa1c80c7ee0 0x7fa1c80c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1c7400640 1 --2- 172.21.15.132:0/4179521550 >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 0x7fa1c80c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1c6a00640 1 -- 172.21.15.132:0/4179521550 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896047137 0 0) 0x7fa1c80c6760 con 0x7fa1c81529d0 2024-03-27T06:22:10.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1cd200640 1 -- 172.21.15.132:0/4179521550 >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 msgr2=0x7fa1c80c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:10.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1cd200640 1 --2- 172.21.15.132:0/4179521550 >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 0x7fa1c80c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1cd200640 1 -- 172.21.15.132:0/4179521550 >> v1:172.21.15.132:6789/0 conn(0x7fa1c81529d0 legacy=0x7fa1c815c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1cd200640 1 -- 172.21.15.132:0/4179521550 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c80c72a0 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.229+0000 7fa1c7400640 1 --2- 172.21.15.132:0/4179521550 >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 0x7fa1c80c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cd200640 1 --2- 172.21.15.132:0/4179521550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c80c8ba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80c7c80 tx=0x7fa1c002be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bc60fec2167607a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1c6a00640 1 -- 172.21.15.132:0/4179521550 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0039070 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1c6a00640 1 -- 172.21.15.132:0/4179521550 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c002cbe0 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1c6a00640 1 -- 172.21.15.132:0/4179521550 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0034440 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 -- 172.21.15.132:0/4179521550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c7ee0 msgr2=0x7fa1c80c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 --2- 172.21.15.132:0/4179521550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c80c8ba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80c7c80 tx=0x7fa1c002be30 comp rx=0 tx=0).stop 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 -- 172.21.15.132:0/4179521550 shutdown_connections 2024-03-27T06:22:10.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 --2- 172.21.15.132:0/4179521550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c80c8ba0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 --2- 172.21.15.132:0/4179521550 >> v2:172.21.15.132:3300/0 conn(0x7fa1c80c3b50 0x7fa1c80c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.231+0000 7fa1cf200640 1 -- 172.21.15.132:0/4179521550 >> 172.21.15.132:0/4179521550 conn(0x7fa1c80cc650 msgr2=0x7fa1c80cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.232+0000 7fa1cf200640 1 -- 172.21.15.132:0/4179521550 shutdown_connections 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.232+0000 7fa1cf200640 1 -- 172.21.15.132:0/4179521550 wait complete. 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.233+0000 7fa1cf200640 1 Processor -- start 2024-03-27T06:22:10.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.233+0000 7fa1cf200640 1 -- start start 2024-03-27T06:22:10.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c3b50 0x7fa1c81e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c81e2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c8152db0 con 0x7fa1c80c7ee0 2024-03-27T06:22:10.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c815bde0 con 0x7fa1c80c3b50 2024-03-27T06:22:10.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cf200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c8067760 con 0x7fa1c81529d0 2024-03-27T06:22:10.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1c7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c3b50 0x7fa1c81e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1cd200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.234+0000 7fa1c7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c81e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:35366/0 (socket says 172.21.15.132:35366) 2024-03-27T06:22:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 -- 172.21.15.132:0/1946949326 learned_addr learned my addr 172.21.15.132:0/1946949326 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:10.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 -- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c3b50 msgr2=0x7fa1c81e37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c3b50 0x7fa1c81e37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 -- 172.21.15.132:0/1946949326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1c80c7ee0 msgr2=0x7fa1c81e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c81e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 -- 172.21.15.132:0/1946949326 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c81e0f80 con 0x7fa1c81529d0 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1cd200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009e80 tx=0x7fa1c0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.235+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0039050 con 0x7fa1c81529d0 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.236+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c0045070 con 0x7fa1c81529d0 2024-03-27T06:22:10.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.236+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c003e830 con 0x7fa1c81529d0 2024-03-27T06:22:10.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.236+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c81560b0 con 0x7fa1c81529d0 2024-03-27T06:22:10.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.236+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c81e1c20 con 0x7fa1c81529d0 2024-03-27T06:22:10.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.237+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c8068820 con 0x7fa1c81529d0 2024-03-27T06:22:10.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.238+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa1c003e9d0 con 0x7fa1c81529d0 2024-03-27T06:22:10.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.238+0000 7fa1bfe00640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 0x7fa1a806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.238+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fa1c00adf10 con 0x7fa1c81529d0 2024-03-27T06:22:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.239+0000 7fa1c7e00640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 0x7fa1a806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.239+0000 7fa1c7e00640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 0x7fa1a806ea20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007fd0 tx=0x7fa1b8007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.243+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fa1c004e020 con 0x7fa1c81529d0 2024-03-27T06:22:10.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.516+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1c806c4f0 con 0x7fa1a806c560 2024-03-27T06:22:10.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.529+0000 7fa1bfe00640 1 -- 172.21.15.132:0/1946949326 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+256708 (secure 0 0 0) 0x7fa1c806c4f0 con 0x7fa1a806c560 2024-03-27T06:22:10.523 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:10.525 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:22:10.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 msgr2=0x7fa1a806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 0x7fa1a806ea20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007fd0 tx=0x7fa1b8007de0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 msgr2=0x7fa1c8149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009e80 tx=0x7fa1c0004a40 comp rx=0 tx=0).stop 2024-03-27T06:22:10.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 shutdown_connections 2024-03-27T06:22:10.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa1c81529d0 0x7fa1c8149d60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1c80c7ee0 0x7fa1c81e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa1a806c560 0x7fa1a806ea20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 --2- 172.21.15.132:0/1946949326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1c80c3b50 0x7fa1c81e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 >> 172.21.15.132:0/1946949326 conn(0x7fa1c80cc650 msgr2=0x7fa1c814c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:10.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 shutdown_connections 2024-03-27T06:22:10.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.535+0000 7fa1cf200640 1 -- 172.21.15.132:0/1946949326 wait complete. 2024-03-27T06:22:10.544 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-03-27T06:22:10.481409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123705486,"num_objects":10754,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10754,"num_whiteouts":0,"num_read":1825,"num_read_kb":514907,"num_write":25292,"num_write_kb":1250430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":95,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25232,"ondisk_log_size":25232,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2114408,"kb_used_data":1755304,"kb_used_omap":12,"kb_used_meta":359027,"kb_avail":752860312,"statfs":{"total":773094113280,"available":770928959488,"internally_reserved":0,"allocated":1797431296,"data_stored":2247640857,"data_compressed":214703604,"data_compressed_allocated":244834304,"data_compressed_original":752640000,"omap_allocated":12751,"internal_metadata":367644209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":129,"apply_latency_ms":129,"commit_latency_ns":129000000,"apply_latency_ns":129000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":124519841,"num_objects":4067,"num_object_clones":0,"num_object_copies":8134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":123710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"3.3f","version":"44'331","reported_seq":2434,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.926354+0000","last_change":"2024-03-27T06:18:40.841294+0000","last_active":"2024-03-27T06:22:02.926354+0000","last_peered":"2024-03-27T06:22:02.926354+0000","last_clean":"2024-03-27T06:22:02.926354+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:22:02.926354+0000","last_undegraded":"2024-03-27T06:22:02.926354+0000","last_fullsized":"2024-03-27T06:22:02.926354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:32:50.072917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048207270000000003,"stat_sum":{"num_bytes":26656138,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":18,"num_read_kb":8207,"num_write":331,"num_write_kb":26201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"44'314","reported_seq":2415,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.638786+0000","last_change":"2024-03-27T06:18:40.829515+0000","last_active":"2024-03-27T06:22:02.638786+0000","last_peered":"2024-03-27T06:22:02.638786+0000","last_clean":"2024-03-27T06:22:02.638786+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:22:02.638786+0000","last_undegraded":"2024-03-27T06:22:02.638786+0000","last_fullsized":"2024-03-27T06:22:02.638786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:20.543774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037350370000000001,"stat_sum":{"num_bytes":5169603,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":315,"num_write_kb":5186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"44'328","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.593021+0000","last_change":"2024-03-27T06:18:40.829898+0000","last_active":"2024-03-27T06:22:02.593021+0000","last_peered":"2024-03-27T06:22:02.593021+0000","last_clean":"2024-03-27T06:22:02.593021+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:22:02.593021+0000","last_undegraded":"2024-03-27T06:22:02.593021+0000","last_fullsized":"2024-03-27T06:22:02.593021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036092289999999998,"stat_sum":{"num_bytes":13816404,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":21,"num_read_kb":8211,"num_write":321,"num_write_kb":21844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"44'308","reported_seq":2407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.859463+0000","last_change":"2024-03-27T06:18:40.562894+0000","last_active":"2024-03-27T06:22:02.859463+0000","last_peered":"2024-03-27T06:22:02.859463+0000","last_clean":"2024-03-27T06:22:02.859463+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:22:02.859463+0000","last_undegraded":"2024-03-27T06:22:02.859463+0000","last_fullsized":"2024-03-27T06:22:02.859463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:29:51.024283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002773321,"stat_sum":{"num_bytes":19633039,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":13,"num_read_kb":8202,"num_write":307,"num_write_kb":19330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"44'350","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.930912+0000","last_change":"2024-03-27T06:18:40.840112+0000","last_active":"2024-03-27T06:22:02.930912+0000","last_peered":"2024-03-27T06:22:02.930912+0000","last_clean":"2024-03-27T06:22:02.930912+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:22:02.930912+0000","last_undegraded":"2024-03-27T06:22:02.930912+0000","last_fullsized":"2024-03-27T06:22:02.930912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:12:12.302147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071357900000000004,"stat_sum":{"num_bytes":5850441,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":350,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"44'320","reported_seq":2425,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.880517+0000","last_change":"2024-03-27T06:18:40.562623+0000","last_active":"2024-03-27T06:22:02.880517+0000","last_peered":"2024-03-27T06:22:02.880517+0000","last_clean":"2024-03-27T06:22:02.880517+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:22:02.880517+0000","last_undegraded":"2024-03-27T06:22:02.880517+0000","last_fullsized":"2024-03-27T06:22:02.880517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:33:31.576614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002466584,"stat_sum":{"num_bytes":14040535,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":20,"num_read_kb":3571,"num_write":321,"num_write_kb":13825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"44'351","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.806369+0000","last_change":"2024-03-27T06:18:40.838197+0000","last_active":"2024-03-27T06:22:02.806369+0000","last_peered":"2024-03-27T06:22:02.806369+0000","last_clean":"2024-03-27T06:22:02.806369+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:22:02.806369+0000","last_undegraded":"2024-03-27T06:22:02.806369+0000","last_fullsized":"2024-03-27T06:22:02.806369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:12:28.032906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056884500000000003,"stat_sum":{"num_bytes":24259091,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":18,"num_read_kb":4113,"num_write":354,"num_write_kb":23807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"44'323","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.745845+0000","last_change":"2024-03-27T06:18:40.830353+0000","last_active":"2024-03-27T06:22:02.745845+0000","last_peered":"2024-03-27T06:22:02.745845+0000","last_clean":"2024-03-27T06:22:02.745845+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:22:02.745845+0000","last_undegraded":"2024-03-27T06:22:02.745845+0000","last_fullsized":"2024-03-27T06:22:02.745845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033930689999999999,"stat_sum":{"num_bytes":15911530,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":12,"num_read_kb":4107,"num_write":326,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"44'320","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.877773+0000","last_change":"2024-03-27T06:18:40.834747+0000","last_active":"2024-03-27T06:22:02.877773+0000","last_peered":"2024-03-27T06:22:02.877773+0000","last_clean":"2024-03-27T06:22:02.877773+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:22:02.877773+0000","last_undegraded":"2024-03-27T06:22:02.877773+0000","last_fullsized":"2024-03-27T06:22:02.877773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:36:36.330080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054729009999999996,"stat_sum":{"num_bytes":6593503,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":20,"num_read_kb":4119,"num_write":318,"num_write_kb":10665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"44'308","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.744529+0000","last_change":"2024-03-27T06:18:40.841044+0000","last_active":"2024-03-27T06:22:02.744529+0000","last_peered":"2024-03-27T06:22:02.744529+0000","last_clean":"2024-03-27T06:22:02.744529+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:22:02.744529+0000","last_undegraded":"2024-03-27T06:22:02.744529+0000","last_fullsized":"2024-03-27T06:22:02.744529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:49.951013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068694350000000001,"stat_sum":{"num_bytes":18722866,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":20,"num_read_kb":9776,"num_write":307,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"44'298","reported_seq":2404,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.851195+0000","last_change":"2024-03-27T06:18:40.840622+0000","last_active":"2024-03-27T06:22:02.851195+0000","last_peered":"2024-03-27T06:22:02.851195+0000","last_clean":"2024-03-27T06:22:02.851195+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:22:02.851195+0000","last_undegraded":"2024-03-27T06:22:02.851195+0000","last_fullsized":"2024-03-27T06:22:02.851195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:17:59.498179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070900370000000004,"stat_sum":{"num_bytes":18171100,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":21,"num_read_kb":8210,"num_write":297,"num_write_kb":17901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"44'297","reported_seq":2391,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.000855+0000","last_change":"2024-03-27T06:18:40.840141+0000","last_active":"2024-03-27T06:22:03.000855+0000","last_peered":"2024-03-27T06:22:03.000855+0000","last_clean":"2024-03-27T06:22:03.000855+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:22:03.000855+0000","last_undegraded":"2024-03-27T06:22:03.000855+0000","last_fullsized":"2024-03-27T06:22:03.000855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:41:13.001519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072900689999999997,"stat_sum":{"num_bytes":8281717,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":296,"num_write_kb":12341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"44'301","reported_seq":2406,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.683907+0000","last_change":"2024-03-27T06:18:40.826940+0000","last_active":"2024-03-27T06:22:08.683907+0000","last_peered":"2024-03-27T06:22:08.683907+0000","last_clean":"2024-03-27T06:22:08.683907+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:22:08.683907+0000","last_undegraded":"2024-03-27T06:22:08.683907+0000","last_fullsized":"2024-03-27T06:22:08.683907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":301,"log_dups_size":0,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029554939999999999,"stat_sum":{"num_bytes":19210972,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":21,"num_read_kb":4116,"num_write":287,"num_write_kb":27094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"44'342","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.653404+0000","last_change":"2024-03-27T06:18:40.827478+0000","last_active":"2024-03-27T06:22:08.653404+0000","last_peered":"2024-03-27T06:22:08.653404+0000","last_clean":"2024-03-27T06:22:08.653404+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:22:08.653404+0000","last_undegraded":"2024-03-27T06:22:08.653404+0000","last_fullsized":"2024-03-27T06:22:08.653404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:34:11.809253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003176613,"stat_sum":{"num_bytes":7395932,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":20,"num_read_kb":4114,"num_write":334,"num_write_kb":11493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.718682+0000","last_change":"2024-03-27T06:18:40.829238+0000","last_active":"2024-03-27T06:22:02.718682+0000","last_peered":"2024-03-27T06:22:02.718682+0000","last_clean":"2024-03-27T06:22:02.718682+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:22:02.718682+0000","last_undegraded":"2024-03-27T06:22:02.718682+0000","last_fullsized":"2024-03-27T06:22:02.718682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:21:38.019283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036973140000000002,"stat_sum":{"num_bytes":23111157,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":349,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"44'349","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.908232+0000","last_change":"2024-03-27T06:18:40.826025+0000","last_active":"2024-03-27T06:22:02.908232+0000","last_peered":"2024-03-27T06:22:02.908232+0000","last_clean":"2024-03-27T06:22:02.908232+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:22:02.908232+0000","last_undegraded":"2024-03-27T06:22:02.908232+0000","last_fullsized":"2024-03-27T06:22:02.908232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001289739,"stat_sum":{"num_bytes":12029261,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":347,"num_write_kb":15208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"44'339","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980167+0000","last_change":"2024-03-27T06:18:40.835662+0000","last_active":"2024-03-27T06:22:02.980167+0000","last_peered":"2024-03-27T06:22:02.980167+0000","last_clean":"2024-03-27T06:22:02.980167+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:22:02.980167+0000","last_undegraded":"2024-03-27T06:22:02.980167+0000","last_fullsized":"2024-03-27T06:22:02.980167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089932469999999994,"stat_sum":{"num_bytes":10346712,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":335,"num_write_kb":14376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"44'313","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.661964+0000","last_change":"2024-03-27T06:18:40.826258+0000","last_active":"2024-03-27T06:22:08.661964+0000","last_peered":"2024-03-27T06:22:08.661964+0000","last_clean":"2024-03-27T06:22:08.661964+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:22:08.661964+0000","last_undegraded":"2024-03-27T06:22:08.661964+0000","last_fullsized":"2024-03-27T06:22:08.661964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:50:38.551582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010652260000000001,"stat_sum":{"num_bytes":8103937,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":302,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"44'319","reported_seq":2426,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.942674+0000","last_change":"2024-03-27T06:18:40.836497+0000","last_active":"2024-03-27T06:22:02.942674+0000","last_peered":"2024-03-27T06:22:02.942674+0000","last_clean":"2024-03-27T06:22:02.942674+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:22:02.942674+0000","last_undegraded":"2024-03-27T06:22:02.942674+0000","last_fullsized":"2024-03-27T06:22:02.942674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:43:03.085499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075917839999999999,"stat_sum":{"num_bytes":14133748,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":23,"num_read_kb":12737,"num_write":315,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"44'345","reported_seq":2453,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.001320+0000","last_change":"2024-03-27T06:18:40.840351+0000","last_active":"2024-03-27T06:22:03.001320+0000","last_peered":"2024-03-27T06:22:03.001320+0000","last_clean":"2024-03-27T06:22:03.001320+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:22:03.001320+0000","last_undegraded":"2024-03-27T06:22:03.001320+0000","last_fullsized":"2024-03-27T06:22:03.001320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:15:43.218527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072459589999999997,"stat_sum":{"num_bytes":23160385,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":24,"num_read_kb":12308,"num_write":342,"num_write_kb":30986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"44'335","reported_seq":2446,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.716055+0000","last_change":"2024-03-27T06:18:40.562416+0000","last_active":"2024-03-27T06:22:02.716055+0000","last_peered":"2024-03-27T06:22:02.716055+0000","last_clean":"2024-03-27T06:22:02.716055+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:22:02.716055+0000","last_undegraded":"2024-03-27T06:22:02.716055+0000","last_fullsized":"2024-03-27T06:22:02.716055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001666102,"stat_sum":{"num_bytes":14662751,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":26,"num_read_kb":4119,"num_write":336,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"44'327","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703008+0000","last_change":"2024-03-27T06:18:40.829281+0000","last_active":"2024-03-27T06:22:02.703008+0000","last_peered":"2024-03-27T06:22:02.703008+0000","last_clean":"2024-03-27T06:22:02.703008+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:22:02.703008+0000","last_undegraded":"2024-03-27T06:22:02.703008+0000","last_fullsized":"2024-03-27T06:22:02.703008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:26:03.545200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024688459999999998,"stat_sum":{"num_bytes":9427787,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":13475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"44'329","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.787322+0000","last_change":"2024-03-27T06:18:40.829763+0000","last_active":"2024-03-27T06:22:02.787322+0000","last_peered":"2024-03-27T06:22:02.787322+0000","last_clean":"2024-03-27T06:22:02.787322+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:22:02.787322+0000","last_undegraded":"2024-03-27T06:22:02.787322+0000","last_fullsized":"2024-03-27T06:22:02.787322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039259519999999999,"stat_sum":{"num_bytes":16419938,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":29,"num_read_kb":18059,"num_write":326,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"44'342","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.905364+0000","last_change":"2024-03-27T06:18:40.826230+0000","last_active":"2024-03-27T06:22:02.905364+0000","last_peered":"2024-03-27T06:22:02.905364+0000","last_clean":"2024-03-27T06:22:02.905364+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:22:02.905364+0000","last_undegraded":"2024-03-27T06:22:02.905364+0000","last_fullsized":"2024-03-27T06:22:02.905364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470573,"stat_sum":{"num_bytes":19403206,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":18,"num_read_kb":8208,"num_write":342,"num_write_kb":19130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"44'316","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730340+0000","last_change":"2024-03-27T06:18:40.829685+0000","last_active":"2024-03-27T06:22:02.730340+0000","last_peered":"2024-03-27T06:22:02.730340+0000","last_clean":"2024-03-27T06:22:02.730340+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:22:02.730340+0000","last_undegraded":"2024-03-27T06:22:02.730340+0000","last_fullsized":"2024-03-27T06:22:02.730340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035867099999999999,"stat_sum":{"num_bytes":14543064,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":314,"num_write_kb":14366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"44'367","reported_seq":2471,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.980468+0000","last_change":"2024-03-27T06:18:40.562911+0000","last_active":"2024-03-27T06:22:02.980468+0000","last_peered":"2024-03-27T06:22:02.980468+0000","last_clean":"2024-03-27T06:22:02.980468+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:22:02.980468+0000","last_undegraded":"2024-03-27T06:22:02.980468+0000","last_fullsized":"2024-03-27T06:22:02.980468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:49:45.282319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002151956,"stat_sum":{"num_bytes":5854740,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":367,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"44'314","reported_seq":2417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.658308+0000","last_change":"2024-03-27T06:18:40.827317+0000","last_active":"2024-03-27T06:22:08.658308+0000","last_peered":"2024-03-27T06:22:08.658308+0000","last_clean":"2024-03-27T06:22:08.658308+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:22:08.658308+0000","last_undegraded":"2024-03-27T06:22:08.658308+0000","last_fullsized":"2024-03-27T06:22:08.658308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:28:40.555644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032490380000000001,"stat_sum":{"num_bytes":14027540,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":18,"num_read_kb":8206,"num_write":307,"num_write_kb":13857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"44'314","reported_seq":2414,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.683280+0000","last_change":"2024-03-27T06:18:40.827238+0000","last_active":"2024-03-27T06:22:08.683280+0000","last_peered":"2024-03-27T06:22:08.683280+0000","last_clean":"2024-03-27T06:22:08.683280+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:22:08.683280+0000","last_undegraded":"2024-03-27T06:22:08.683280+0000","last_fullsized":"2024-03-27T06:22:08.683280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:35:39.617870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003068571,"stat_sum":{"num_bytes":18001138,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":16,"num_read_kb":7383,"num_write":307,"num_write_kb":21016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"44'351","reported_seq":2458,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.617101+0000","last_change":"2024-03-27T06:18:40.829449+0000","last_active":"2024-03-27T06:22:02.617101+0000","last_peered":"2024-03-27T06:22:02.617101+0000","last_clean":"2024-03-27T06:22:02.617101+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:22:02.617101+0000","last_undegraded":"2024-03-27T06:22:02.617101+0000","last_fullsized":"2024-03-27T06:22:02.617101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:47:20.901338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036751610000000001,"stat_sum":{"num_bytes":22534890,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":22,"num_read_kb":12306,"num_write":351,"num_write_kb":22191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"44'351","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.911800+0000","last_change":"2024-03-27T06:18:40.836714+0000","last_active":"2024-03-27T06:22:02.911800+0000","last_peered":"2024-03-27T06:22:02.911800+0000","last_clean":"2024-03-27T06:22:02.911800+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:22:02.911800+0000","last_undegraded":"2024-03-27T06:22:02.911800+0000","last_fullsized":"2024-03-27T06:22:02.911800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044159999999999,"stat_sum":{"num_bytes":14250100,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":24,"num_read_kb":7593,"num_write":351,"num_write_kb":14104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"44'387","reported_seq":2491,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671810+0000","last_change":"2024-03-27T06:18:40.825940+0000","last_active":"2024-03-27T06:22:08.671810+0000","last_peered":"2024-03-27T06:22:08.671810+0000","last_clean":"2024-03-27T06:22:08.671810+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:22:08.671810+0000","last_undegraded":"2024-03-27T06:22:08.671810+0000","last_fullsized":"2024-03-27T06:22:08.671810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:54:24.095203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011795639999999999,"stat_sum":{"num_bytes":10303903,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":19,"num_read_kb":1482,"num_write":377,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"44'371","reported_seq":2484,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.738005+0000","last_change":"2024-03-27T06:18:40.561817+0000","last_active":"2024-03-27T06:22:02.738005+0000","last_peered":"2024-03-27T06:22:02.738005+0000","last_clean":"2024-03-27T06:22:02.738005+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:22:02.738005+0000","last_undegraded":"2024-03-27T06:22:02.738005+0000","last_fullsized":"2024-03-27T06:22:02.738005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:57:04.359511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016911179999999999,"stat_sum":{"num_bytes":11109721,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":28,"num_read_kb":4123,"num_write":370,"num_write_kb":11044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"44'320","reported_seq":2423,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.653651+0000","last_change":"2024-03-27T06:18:40.831829+0000","last_active":"2024-03-27T06:22:02.653651+0000","last_peered":"2024-03-27T06:22:02.653651+0000","last_clean":"2024-03-27T06:22:02.653651+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:22:02.653651+0000","last_undegraded":"2024-03-27T06:22:02.653651+0000","last_fullsized":"2024-03-27T06:22:02.653651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:18:03.157765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057058689999999997,"stat_sum":{"num_bytes":18559363,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":18,"num_read_kb":4112,"num_write":320,"num_write_kb":18284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"43'17","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283493+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:21:30.283493+0000","last_peered":"2024-03-27T06:21:30.283493+0000","last_clean":"2024-03-27T06:21:30.283493+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:21:30.283493+0000","last_undegraded":"2024-03-27T06:21:30.283493+0000","last_fullsized":"2024-03-27T06:21:30.283493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195296,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":18,"num_write_kb":4248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'311","reported_seq":2413,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.773731+0000","last_change":"2024-03-27T06:18:40.562829+0000","last_active":"2024-03-27T06:22:02.773731+0000","last_peered":"2024-03-27T06:22:02.773731+0000","last_clean":"2024-03-27T06:22:02.773731+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:22:02.773731+0000","last_undegraded":"2024-03-27T06:22:02.773731+0000","last_fullsized":"2024-03-27T06:22:02.773731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:17:17.674549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018454000000000001,"stat_sum":{"num_bytes":6743271,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":310,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"44'253","reported_seq":326,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.246658+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:21:55.246658+0000","last_peered":"2024-03-27T06:21:55.246658+0000","last_clean":"2024-03-27T06:21:55.246658+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:21:55.246658+0000","last_undegraded":"2024-03-27T06:21:55.246658+0000","last_fullsized":"2024-03-27T06:21:55.246658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":8577,"num_write":252,"num_write_kb":12472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"44'63","reported_seq":133,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250279+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:21:55.250279+0000","last_peered":"2024-03-27T06:21:55.250279+0000","last_clean":"2024-03-27T06:21:55.250279+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:21:55.250279+0000","last_undegraded":"2024-03-27T06:21:55.250279+0000","last_fullsized":"2024-03-27T06:21:55.250279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":65,"num_write_kb":4173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"44'399","reported_seq":2511,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.700284+0000","last_change":"2024-03-27T06:18:40.830513+0000","last_active":"2024-03-27T06:22:02.700284+0000","last_peered":"2024-03-27T06:22:02.700284+0000","last_clean":"2024-03-27T06:22:02.700284+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:22:02.700284+0000","last_undegraded":"2024-03-27T06:22:02.700284+0000","last_fullsized":"2024-03-27T06:22:02.700284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003103719,"stat_sum":{"num_bytes":18931276,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":28,"num_read_kb":12930,"num_write":396,"num_write_kb":26887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"44'377","reported_seq":442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:53.574559+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:21:53.574559+0000","last_peered":"2024-03-27T06:21:53.574559+0000","last_clean":"2024-03-27T06:21:53.574559+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:21:53.574559+0000","last_undegraded":"2024-03-27T06:21:53.574559+0000","last_fullsized":"2024-03-27T06:21:53.574559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:20:55.700931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":374,"num_write_kb":12507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"44'335","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.674076+0000","last_change":"2024-03-27T06:18:40.562924+0000","last_active":"2024-03-27T06:22:02.674076+0000","last_peered":"2024-03-27T06:22:02.674076+0000","last_clean":"2024-03-27T06:22:02.674076+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:22:02.674076+0000","last_undegraded":"2024-03-27T06:22:02.674076+0000","last_fullsized":"2024-03-27T06:22:02.674076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:17:24.783114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019163050000000001,"stat_sum":{"num_bytes":19664158,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":13845,"num_write":329,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"43'10","reported_seq":76,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.401768+0000","last_change":"2024-03-27T06:14:02.527835+0000","last_active":"2024-03-27T06:21:30.401768+0000","last_peered":"2024-03-27T06:21:30.401768+0000","last_clean":"2024-03-27T06:21:30.401768+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:21:30.401768+0000","last_undegraded":"2024-03-27T06:21:30.401768+0000","last_fullsized":"2024-03-27T06:21:30.401768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"44'351","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.986843+0000","last_change":"2024-03-27T06:18:40.563693+0000","last_active":"2024-03-27T06:22:02.986843+0000","last_peered":"2024-03-27T06:22:02.986843+0000","last_clean":"2024-03-27T06:22:02.986843+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:22:02.986843+0000","last_undegraded":"2024-03-27T06:22:02.986843+0000","last_fullsized":"2024-03-27T06:22:02.986843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026075790000000001,"stat_sum":{"num_bytes":22946854,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":25,"num_read_kb":12308,"num_write":351,"num_write_kb":22593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"44'107","reported_seq":173,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.610595+0000","last_change":"2024-03-27T06:14:02.529280+0000","last_active":"2024-03-27T06:21:29.610595+0000","last_peered":"2024-03-27T06:21:29.610595+0000","last_clean":"2024-03-27T06:21:29.610595+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:21:29.610595+0000","last_undegraded":"2024-03-27T06:21:29.610595+0000","last_fullsized":"2024-03-27T06:21:29.610595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":106,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"44'320","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.808658+0000","last_change":"2024-03-27T06:18:40.830168+0000","last_active":"2024-03-27T06:22:02.808658+0000","last_peered":"2024-03-27T06:22:02.808658+0000","last_clean":"2024-03-27T06:22:02.808658+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:22:02.808658+0000","last_undegraded":"2024-03-27T06:22:02.808658+0000","last_fullsized":"2024-03-27T06:22:02.808658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003650304,"stat_sum":{"num_bytes":10082281,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":22,"num_read_kb":4115,"num_write":319,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"44'282","reported_seq":348,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.729600+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:22:08.729600+0000","last_peered":"2024-03-27T06:22:08.729600+0000","last_clean":"2024-03-27T06:22:08.729600+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:22:08.729600+0000","last_undegraded":"2024-03-27T06:22:08.729600+0000","last_fullsized":"2024-03-27T06:22:08.729600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":284,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'315","reported_seq":2416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.750367+0000","last_change":"2024-03-27T06:18:40.835973+0000","last_active":"2024-03-27T06:22:02.750367+0000","last_peered":"2024-03-27T06:22:02.750367+0000","last_clean":"2024-03-27T06:22:02.750367+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:22:02.750367+0000","last_undegraded":"2024-03-27T06:22:02.750367+0000","last_fullsized":"2024-03-27T06:22:02.750367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084315699999999993,"stat_sum":{"num_bytes":18808790,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":315,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"44'127","reported_seq":197,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:48.532269+0000","last_change":"2024-03-27T06:14:02.516337+0000","last_active":"2024-03-27T06:21:48.532269+0000","last_peered":"2024-03-27T06:21:48.532269+0000","last_clean":"2024-03-27T06:21:48.532269+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:21:48.532269+0000","last_undegraded":"2024-03-27T06:21:48.532269+0000","last_fullsized":"2024-03-27T06:21:48.532269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":8203,"num_write":123,"num_write_kb":12364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"44'327","reported_seq":2430,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.742161+0000","last_change":"2024-03-27T06:18:40.841177+0000","last_active":"2024-03-27T06:22:02.742161+0000","last_peered":"2024-03-27T06:22:02.742161+0000","last_clean":"2024-03-27T06:22:02.742161+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:22:02.742161+0000","last_undegraded":"2024-03-27T06:22:02.742161+0000","last_fullsized":"2024-03-27T06:22:02.742161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:44:07.500058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050386370000000003,"stat_sum":{"num_bytes":2468399,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":327,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"44'145","reported_seq":220,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978468+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:22:03.978468+0000","last_peered":"2024-03-27T06:22:03.978468+0000","last_clean":"2024-03-27T06:22:03.978468+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:22:03.978468+0000","last_undegraded":"2024-03-27T06:22:03.978468+0000","last_fullsized":"2024-03-27T06:22:03.978468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":8224,"num_write":147,"num_write_kb":8324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'350","reported_seq":2459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.673505+0000","last_change":"2024-03-27T06:18:40.826892+0000","last_active":"2024-03-27T06:22:08.673505+0000","last_peered":"2024-03-27T06:22:08.673505+0000","last_clean":"2024-03-27T06:22:08.673505+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:22:08.673505+0000","last_undegraded":"2024-03-27T06:22:08.673505+0000","last_fullsized":"2024-03-27T06:22:08.673505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002905866,"stat_sum":{"num_bytes":19666163,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":25,"num_read_kb":16403,"num_write":336,"num_write_kb":27577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"44'6","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.214987+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:21:29.214987+0000","last_peered":"2024-03-27T06:21:29.214987+0000","last_clean":"2024-03-27T06:21:29.214987+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:21:29.214987+0000","last_undegraded":"2024-03-27T06:21:29.214987+0000","last_fullsized":"2024-03-27T06:21:29.214987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"44'346","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.820209+0000","last_change":"2024-03-27T06:18:40.837185+0000","last_active":"2024-03-27T06:22:02.820209+0000","last_peered":"2024-03-27T06:22:02.820209+0000","last_clean":"2024-03-27T06:22:02.820209+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:22:02.820209+0000","last_undegraded":"2024-03-27T06:22:02.820209+0000","last_fullsized":"2024-03-27T06:22:02.820209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010165321999999999,"stat_sum":{"num_bytes":10617237,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":345,"num_write_kb":10547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"44'12","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981540+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:22:03.981540+0000","last_peered":"2024-03-27T06:22:03.981540+0000","last_clean":"2024-03-27T06:22:03.981540+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:22:03.981540+0000","last_undegraded":"2024-03-27T06:22:03.981540+0000","last_fullsized":"2024-03-27T06:22:03.981540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":11,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"44'347","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.588028+0000","last_change":"2024-03-27T06:18:40.831916+0000","last_active":"2024-03-27T06:22:02.588028+0000","last_peered":"2024-03-27T06:22:02.588028+0000","last_clean":"2024-03-27T06:22:02.588028+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:22:02.588028+0000","last_undegraded":"2024-03-27T06:22:02.588028+0000","last_fullsized":"2024-03-27T06:22:02.588028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:33:54.035813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054212890000000001,"stat_sum":{"num_bytes":24812922,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":17,"num_read_kb":8206,"num_write":342,"num_write_kb":26402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"44'278","reported_seq":359,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250589+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:21:55.250589+0000","last_peered":"2024-03-27T06:21:55.250589+0000","last_clean":"2024-03-27T06:21:55.250589+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:21:55.250589+0000","last_undegraded":"2024-03-27T06:21:55.250589+0000","last_fullsized":"2024-03-27T06:21:55.250589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":283,"num_write_kb":8391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"44'358","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.826841+0000","last_change":"2024-03-27T06:18:40.834701+0000","last_active":"2024-03-27T06:22:02.826841+0000","last_peered":"2024-03-27T06:22:02.826841+0000","last_clean":"2024-03-27T06:22:02.826841+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:22:02.826841+0000","last_undegraded":"2024-03-27T06:22:02.826841+0000","last_fullsized":"2024-03-27T06:22:02.826841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083622499999999999,"stat_sum":{"num_bytes":12063538,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":20,"num_read_kb":12305,"num_write":353,"num_write_kb":20111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"44'29","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212511+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:21:34.212511+0000","last_peered":"2024-03-27T06:21:34.212511+0000","last_clean":"2024-03-27T06:21:34.212511+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:21:34.212511+0000","last_undegraded":"2024-03-27T06:21:34.212511+0000","last_fullsized":"2024-03-27T06:21:34.212511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4103,"num_write":34,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'291","reported_seq":2389,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.989310+0000","last_change":"2024-03-27T06:18:40.563367+0000","last_active":"2024-03-27T06:22:02.989310+0000","last_peered":"2024-03-27T06:22:02.989310+0000","last_clean":"2024-03-27T06:22:02.989310+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:22:02.989310+0000","last_undegraded":"2024-03-27T06:22:02.989310+0000","last_fullsized":"2024-03-27T06:22:02.989310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022904700000000002,"stat_sum":{"num_bytes":11066744,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":12134,"num_write":288,"num_write_kb":15054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"44'28","reported_seq":93,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601016+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:21:29.601016+0000","last_peered":"2024-03-27T06:21:29.601016+0000","last_clean":"2024-03-27T06:21:29.601016+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:21:29.601016+0000","last_undegraded":"2024-03-27T06:21:29.601016+0000","last_fullsized":"2024-03-27T06:21:29.601016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":30,"num_write_kb":4134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"44'327","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.837377+0000","last_change":"2024-03-27T06:18:40.841709+0000","last_active":"2024-03-27T06:22:02.837377+0000","last_peered":"2024-03-27T06:22:02.837377+0000","last_clean":"2024-03-27T06:22:02.837377+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:22:02.837377+0000","last_undegraded":"2024-03-27T06:22:02.837377+0000","last_fullsized":"2024-03-27T06:22:02.837377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:43:54.568781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050322370000000002,"stat_sum":{"num_bytes":14587209,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":24,"num_read_kb":12307,"num_write":322,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"44'23","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283694+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:21:30.283694+0000","last_peered":"2024-03-27T06:21:30.283694+0000","last_clean":"2024-03-27T06:21:30.283694+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:21:30.283694+0000","last_undegraded":"2024-03-27T06:21:30.283694+0000","last_fullsized":"2024-03-27T06:21:30.283694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":23,"num_write_kb":4120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"44'356","reported_seq":2457,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.998863+0000","last_change":"2024-03-27T06:18:40.840461+0000","last_active":"2024-03-27T06:22:02.998863+0000","last_peered":"2024-03-27T06:22:02.998863+0000","last_clean":"2024-03-27T06:22:02.998863+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:22:02.998863+0000","last_undegraded":"2024-03-27T06:22:02.998863+0000","last_fullsized":"2024-03-27T06:22:02.998863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:44:35.295705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070052669999999999,"stat_sum":{"num_bytes":11205183,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":352,"num_write_kb":11129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"44'29","reported_seq":101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:58.564457+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:21:58.564457+0000","last_peered":"2024-03-27T06:21:58.564457+0000","last_clean":"2024-03-27T06:21:58.564457+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:21:58.564457+0000","last_undegraded":"2024-03-27T06:21:58.564457+0000","last_fullsized":"2024-03-27T06:21:58.564457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":32,"num_write_kb":4138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"44'323","reported_seq":2427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.968996+0000","last_change":"2024-03-27T06:18:40.835886+0000","last_active":"2024-03-27T06:22:02.968996+0000","last_peered":"2024-03-27T06:22:02.968996+0000","last_clean":"2024-03-27T06:22:02.968996+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:22:02.968996+0000","last_undegraded":"2024-03-27T06:22:02.968996+0000","last_fullsized":"2024-03-27T06:22:02.968996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:18:41.783945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086058899999999997,"stat_sum":{"num_bytes":14608246,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":19,"num_read_kb":4113,"num_write":322,"num_write_kb":18530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.384769+0000","last_change":"2024-03-27T06:13:58.726844+0000","last_active":"2024-03-27T06:21:30.384769+0000","last_peered":"2024-03-27T06:21:30.384769+0000","last_clean":"2024-03-27T06:21:30.384769+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:21:30.384769+0000","last_undegraded":"2024-03-27T06:21:30.384769+0000","last_fullsized":"2024-03-27T06:21:30.384769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:13:57.385319+0000","last_clean_scrub_stamp":"2024-03-27T06:13:57.385319+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:22.958598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'84","reported_seq":152,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.212620+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:21:34.212620+0000","last_peered":"2024-03-27T06:21:34.212620+0000","last_clean":"2024-03-27T06:21:34.212620+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:21:34.212620+0000","last_undegraded":"2024-03-27T06:21:34.212620+0000","last_fullsized":"2024-03-27T06:21:34.212620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":86,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'321","reported_seq":2420,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.778867+0000","last_change":"2024-03-27T06:18:40.563107+0000","last_active":"2024-03-27T06:22:02.778867+0000","last_peered":"2024-03-27T06:22:02.778867+0000","last_clean":"2024-03-27T06:22:02.778867+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:22:02.778867+0000","last_undegraded":"2024-03-27T06:22:02.778867+0000","last_fullsized":"2024-03-27T06:22:02.778867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:37:09.221434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002005691,"stat_sum":{"num_bytes":10118185,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":14,"num_read_kb":4109,"num_write":319,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"44'228","reported_seq":300,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.247793+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:21:55.247793+0000","last_peered":"2024-03-27T06:21:55.247793+0000","last_clean":"2024-03-27T06:21:55.247793+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:21:55.247793+0000","last_undegraded":"2024-03-27T06:21:55.247793+0000","last_fullsized":"2024-03-27T06:21:55.247793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:42:28.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":8210,"num_write":225,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'331","reported_seq":2426,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.660633+0000","last_change":"2024-03-27T06:18:40.825563+0000","last_active":"2024-03-27T06:22:08.660633+0000","last_peered":"2024-03-27T06:22:08.660633+0000","last_clean":"2024-03-27T06:22:08.660633+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:22:08.660633+0000","last_undegraded":"2024-03-27T06:22:08.660633+0000","last_fullsized":"2024-03-27T06:22:08.660633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001779484,"stat_sum":{"num_bytes":10043455,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":324,"num_write_kb":9981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"44'91","reported_seq":242,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.249897+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:21:55.249897+0000","last_peered":"2024-03-27T06:21:55.249897+0000","last_clean":"2024-03-27T06:21:55.249897+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:21:55.249897+0000","last_undegraded":"2024-03-27T06:21:55.249897+0000","last_fullsized":"2024-03-27T06:21:55.249897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534287,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":595,"num_write":90,"num_write_kb":602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"44'332","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.703670+0000","last_change":"2024-03-27T06:18:40.830316+0000","last_active":"2024-03-27T06:22:02.703670+0000","last_peered":"2024-03-27T06:22:02.703670+0000","last_clean":"2024-03-27T06:22:02.703670+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:22:02.703670+0000","last_undegraded":"2024-03-27T06:22:02.703670+0000","last_fullsized":"2024-03-27T06:22:02.703670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031179929999999999,"stat_sum":{"num_bytes":2779321,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":332,"num_write_kb":2894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"44'41","reported_seq":106,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:34.216718+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:21:34.216718+0000","last_peered":"2024-03-27T06:21:34.216718+0000","last_clean":"2024-03-27T06:21:34.216718+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:21:34.216718+0000","last_undegraded":"2024-03-27T06:21:34.216718+0000","last_fullsized":"2024-03-27T06:21:34.216718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:17.040789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":36,"num_write_kb":8227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"44'353","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.665718+0000","last_change":"2024-03-27T06:18:40.826852+0000","last_active":"2024-03-27T06:22:08.665718+0000","last_peered":"2024-03-27T06:22:08.665718+0000","last_clean":"2024-03-27T06:22:08.665718+0000","last_became_active":"2024-03-27T06:14:49.853879+0000","last_became_peered":"2024-03-27T06:14:49.853879+0000","last_unstale":"2024-03-27T06:22:08.665718+0000","last_undegraded":"2024-03-27T06:22:08.665718+0000","last_fullsized":"2024-03-27T06:22:08.665718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:54:30.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022092909999999999,"stat_sum":{"num_bytes":5308376,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":346,"num_write_kb":5365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"44'78","reported_seq":141,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:37.150979+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:21:37.150979+0000","last_peered":"2024-03-27T06:21:37.150979+0000","last_clean":"2024-03-27T06:21:37.150979+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:21:37.150979+0000","last_undegraded":"2024-03-27T06:21:37.150979+0000","last_fullsized":"2024-03-27T06:21:37.150979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":4150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"44'340","reported_seq":2448,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.631346+0000","last_change":"2024-03-27T06:18:40.563208+0000","last_active":"2024-03-27T06:22:02.631346+0000","last_peered":"2024-03-27T06:22:02.631346+0000","last_clean":"2024-03-27T06:22:02.631346+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:22:02.631346+0000","last_undegraded":"2024-03-27T06:22:02.631346+0000","last_fullsized":"2024-03-27T06:22:02.631346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002460744,"stat_sum":{"num_bytes":18473602,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":23,"num_read_kb":12305,"num_write":340,"num_write_kb":22314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'587","reported_seq":652,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:40.150130+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:21:40.150130+0000","last_peered":"2024-03-27T06:21:40.150130+0000","last_clean":"2024-03-27T06:21:40.150130+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:21:40.150130+0000","last_undegraded":"2024-03-27T06:21:40.150130+0000","last_fullsized":"2024-03-27T06:21:40.150130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:54.018683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":587,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"44'340","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671368+0000","last_change":"2024-03-27T06:18:40.827659+0000","last_active":"2024-03-27T06:22:08.671368+0000","last_peered":"2024-03-27T06:22:08.671368+0000","last_clean":"2024-03-27T06:22:08.671368+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:22:08.671368+0000","last_undegraded":"2024-03-27T06:22:08.671368+0000","last_fullsized":"2024-03-27T06:22:08.671368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032966190000000002,"stat_sum":{"num_bytes":6064397,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":331,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"44'137","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.629820+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:22:03.629820+0000","last_peered":"2024-03-27T06:22:03.629820+0000","last_clean":"2024-03-27T06:22:03.629820+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:22:03.629820+0000","last_undegraded":"2024-03-27T06:22:03.629820+0000","last_fullsized":"2024-03-27T06:22:03.629820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":322,"num_write_kb":2012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"44'317","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.566427+0000","last_change":"2024-03-27T06:18:40.562700+0000","last_active":"2024-03-27T06:22:02.566427+0000","last_peered":"2024-03-27T06:22:02.566427+0000","last_clean":"2024-03-27T06:22:02.566427+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:22:02.566427+0000","last_undegraded":"2024-03-27T06:22:02.566427+0000","last_fullsized":"2024-03-27T06:22:02.566427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:57:15.843788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017608820000000001,"stat_sum":{"num_bytes":20225923,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":318,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"43'79","reported_seq":145,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.215719+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:21:29.215719+0000","last_peered":"2024-03-27T06:21:29.215719+0000","last_clean":"2024-03-27T06:21:29.215719+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:21:29.215719+0000","last_undegraded":"2024-03-27T06:21:29.215719+0000","last_fullsized":"2024-03-27T06:21:29.215719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:51:25.292128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":78,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'350","reported_seq":2469,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.556934+0000","last_change":"2024-03-27T06:18:40.562997+0000","last_active":"2024-03-27T06:22:02.556934+0000","last_peered":"2024-03-27T06:22:02.556934+0000","last_clean":"2024-03-27T06:22:02.556934+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:22:02.556934+0000","last_undegraded":"2024-03-27T06:22:02.556934+0000","last_fullsized":"2024-03-27T06:22:02.556934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:19:49.684960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018869889999999999,"stat_sum":{"num_bytes":32233430,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":35,"num_read_kb":18086,"num_write":345,"num_write_kb":31650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"44'31","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.609369+0000","last_change":"2024-03-27T06:14:02.527715+0000","last_active":"2024-03-27T06:21:29.609369+0000","last_peered":"2024-03-27T06:21:29.609369+0000","last_clean":"2024-03-27T06:21:29.609369+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:21:29.609369+0000","last_undegraded":"2024-03-27T06:21:29.609369+0000","last_fullsized":"2024-03-27T06:21:29.609369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":35,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'347","reported_seq":2451,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.890096+0000","last_change":"2024-03-27T06:18:40.836137+0000","last_active":"2024-03-27T06:22:02.890096+0000","last_peered":"2024-03-27T06:22:02.890096+0000","last_clean":"2024-03-27T06:22:02.890096+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:22:02.890096+0000","last_undegraded":"2024-03-27T06:22:02.890096+0000","last_fullsized":"2024-03-27T06:22:02.890096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081323179999999995,"stat_sum":{"num_bytes":22949808,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":346,"num_write_kb":22595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"44'118","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601177+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:21:29.601177+0000","last_peered":"2024-03-27T06:21:29.601177+0000","last_clean":"2024-03-27T06:21:29.601177+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:21:29.601177+0000","last_undegraded":"2024-03-27T06:21:29.601177+0000","last_fullsized":"2024-03-27T06:21:29.601177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":120,"num_write_kb":4433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"44'355","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.550157+0000","last_change":"2024-03-27T06:18:40.562590+0000","last_active":"2024-03-27T06:22:02.550157+0000","last_peered":"2024-03-27T06:22:02.550157+0000","last_clean":"2024-03-27T06:22:02.550157+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:22:02.550157+0000","last_undegraded":"2024-03-27T06:22:02.550157+0000","last_fullsized":"2024-03-27T06:22:02.550157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:13:24.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001751964,"stat_sum":{"num_bytes":19413938,"num_objects":185,"num_object_clones":0,"num_object_copies":370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":21,"num_read_kb":12304,"num_write":357,"num_write_kb":23191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"44'27","reported_seq":92,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.980288+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:22:03.980288+0000","last_peered":"2024-03-27T06:22:03.980288+0000","last_clean":"2024-03-27T06:22:03.980288+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:22:03.980288+0000","last_undegraded":"2024-03-27T06:22:03.980288+0000","last_fullsized":"2024-03-27T06:22:03.980288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":4098,"num_write":32,"num_write_kb":4123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'327","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.665398+0000","last_change":"2024-03-27T06:18:40.826508+0000","last_active":"2024-03-27T06:22:08.665398+0000","last_peered":"2024-03-27T06:22:08.665398+0000","last_clean":"2024-03-27T06:22:08.665398+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:22:08.665398+0000","last_undegraded":"2024-03-27T06:22:08.665398+0000","last_fullsized":"2024-03-27T06:22:08.665398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:40.745990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001518585,"stat_sum":{"num_bytes":15553910,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":17,"num_read_kb":5089,"num_write":319,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"44'200","reported_seq":268,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981086+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:22:03.981086+0000","last_peered":"2024-03-27T06:22:03.981086+0000","last_clean":"2024-03-27T06:22:03.981086+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:22:03.981086+0000","last_undegraded":"2024-03-27T06:22:03.981086+0000","last_fullsized":"2024-03-27T06:22:03.981086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:49.901503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9715716,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":197,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'347","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.917806+0000","last_change":"2024-03-27T06:18:40.836916+0000","last_active":"2024-03-27T06:22:02.917806+0000","last_peered":"2024-03-27T06:22:02.917806+0000","last_clean":"2024-03-27T06:22:02.917806+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:22:02.917806+0000","last_undegraded":"2024-03-27T06:22:02.917806+0000","last_fullsized":"2024-03-27T06:22:02.917806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:26:09.736687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044719999999999,"stat_sum":{"num_bytes":10744528,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":347,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'307","reported_seq":2417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.975860+0000","last_change":"2024-03-27T06:18:40.835719+0000","last_active":"2024-03-27T06:22:02.975860+0000","last_peered":"2024-03-27T06:22:02.975860+0000","last_clean":"2024-03-27T06:22:02.975860+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:22:02.975860+0000","last_undegraded":"2024-03-27T06:22:02.975860+0000","last_fullsized":"2024-03-27T06:22:02.975860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086575760000000002,"stat_sum":{"num_bytes":8008268,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4119,"num_write":307,"num_write_kb":12040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"30'22","reported_seq":87,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.611265+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:21:29.611265+0000","last_peered":"2024-03-27T06:21:29.611265+0000","last_clean":"2024-03-27T06:21:29.611265+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:21:29.611265+0000","last_undegraded":"2024-03-27T06:21:29.611265+0000","last_fullsized":"2024-03-27T06:21:29.611265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:34:41.417294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":22,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"44'335","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.726938+0000","last_change":"2024-03-27T06:18:40.835158+0000","last_active":"2024-03-27T06:22:02.726938+0000","last_peered":"2024-03-27T06:22:02.726938+0000","last_clean":"2024-03-27T06:22:02.726938+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:22:02.726938+0000","last_undegraded":"2024-03-27T06:22:02.726938+0000","last_fullsized":"2024-03-27T06:22:02.726938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:10:12.399076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061971559999999997,"stat_sum":{"num_bytes":27398907,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":15,"num_read_kb":8205,"num_write":334,"num_write_kb":31029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"44'55","reported_seq":125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.239947+0000","last_change":"2024-03-27T06:14:02.533713+0000","last_active":"2024-03-27T06:21:55.239947+0000","last_peered":"2024-03-27T06:21:55.239947+0000","last_clean":"2024-03-27T06:21:55.239947+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:21:55.239947+0000","last_undegraded":"2024-03-27T06:21:55.239947+0000","last_fullsized":"2024-03-27T06:21:55.239947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":12312,"num_write":46,"num_write_kb":12316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"44'312","reported_seq":2418,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.730202+0000","last_change":"2024-03-27T06:18:40.829954+0000","last_active":"2024-03-27T06:22:02.730202+0000","last_peered":"2024-03-27T06:22:02.730202+0000","last_clean":"2024-03-27T06:22:02.730202+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:22:02.730202+0000","last_undegraded":"2024-03-27T06:22:02.730202+0000","last_fullsized":"2024-03-27T06:22:02.730202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036426230000000002,"stat_sum":{"num_bytes":14422279,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":21,"num_read_kb":4923,"num_write":311,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"44'81","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978121+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:22:03.978121+0000","last_peered":"2024-03-27T06:22:03.978121+0000","last_clean":"2024-03-27T06:22:03.978121+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:22:03.978121+0000","last_undegraded":"2024-03-27T06:22:03.978121+0000","last_fullsized":"2024-03-27T06:22:03.978121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":75,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"44'342","reported_seq":2442,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:02.795505+0000","last_change":"2024-03-27T06:18:40.830116+0000","last_active":"2024-03-27T06:22:02.795505+0000","last_peered":"2024-03-27T06:22:02.795505+0000","last_clean":"2024-03-27T06:22:02.795505+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:22:02.795505+0000","last_undegraded":"2024-03-27T06:22:02.795505+0000","last_fullsized":"2024-03-27T06:22:02.795505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036711629999999999,"stat_sum":{"num_bytes":15186793,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":8204,"num_write":343,"num_write_kb":14950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"44'129","reported_seq":207,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.980283+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:22:03.980283+0000","last_peered":"2024-03-27T06:22:03.980283+0000","last_clean":"2024-03-27T06:22:03.980283+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:22:03.980283+0000","last_undegraded":"2024-03-27T06:22:03.980283+0000","last_fullsized":"2024-03-27T06:22:03.980283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":4136,"num_write":129,"num_write_kb":8304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940893603,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":1256,"num_read_kb":394723,"num_write":21174,"num_write_kb":1064721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1712472064,"data_stored":1880247110,"data_compressed":164906076,"data_compressed_allocated":182468608,"data_compressed_original":405446656,"omap_allocated":0,"internal_metadata":0},"log_size":21341,"ondisk_log_size":21341,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182221515,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":453,"num_read_kb":120084,"num_write":3959,"num_write_kb":183293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":95,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":80830464,"data_stored":364443171,"data_compressed":49732790,"data_compressed_allocated":62275584,"data_compressed_original":346046464,"omap_allocated":0,"internal_metadata":0},"log_size":3774,"ondisk_log_size":3774,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170540,"kb_used_data":143524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94201300,"statfs":{"total":96636764160,"available":96462131200,"internally_reserved":0,"allocated":146968576,"data_stored":174700599,"data_compressed":11348325,"data_compressed_allocated":13275136,"data_compressed_original":46198784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196900,"kb_used_data":169884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94174940,"statfs":{"total":96636764160,"available":96435138560,"internally_reserved":0,"allocated":173961216,"data_stored":215859568,"data_compressed":22390241,"data_compressed_allocated":25247744,"data_compressed_original":73641984,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291908,"kb_used_data":242684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94079932,"statfs":{"total":96636764160,"available":96337850368,"internally_reserved":0,"allocated":248508416,"data_stored":304860446,"data_compressed":28268738,"data_compressed_allocated":32043008,"data_compressed_original":96636928,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266772,"kb_used_data":217548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94105068,"statfs":{"total":96636764160,"available":96363589632,"internally_reserved":0,"allocated":222769152,"data_stored":282203639,"data_compressed":27135224,"data_compressed_allocated":30642176,"data_compressed_original":95821824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313816,"kb_used_data":264592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94058024,"statfs":{"total":96636764160,"available":96315416576,"internally_reserved":0,"allocated":270942208,"data_stored":358995823,"data_compressed":39540899,"data_compressed_allocated":45490176,"data_compressed_original":142065664,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267308,"kb_used_data":222244,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94104532,"statfs":{"total":96636764160,"available":96363040768,"internally_reserved":0,"allocated":227577856,"data_stored":275207385,"data_compressed":26559283,"data_compressed_allocated":30052352,"data_compressed_original":85663744,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246944,"kb_used_data":201880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94124896,"statfs":{"total":96636764160,"available":96383893504,"internally_reserved":0,"allocated":206725120,"data_stored":255586545,"data_compressed":23288424,"data_compressed_allocated":26611712,"data_compressed_original":81715200,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360220,"kb_used_data":292948,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94011620,"statfs":{"total":96636764160,"available":96267898880,"internally_reserved":0,"allocated":299978752,"data_stored":380226852,"data_compressed":36172470,"data_compressed_allocated":41472000,"data_compressed_original":130895872,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":67110666,"data_compressed":8915435,"data_compressed_allocated":11333632,"data_compressed_original":64311296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7794688,"data_stored":37748896,"data_compressed":4858381,"data_compressed_allocated":6283264,"data_compressed_original":36245504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6713344,"data_stored":30687942,"data_compressed":4449598,"data_compressed_allocated":5500928,"data_compressed_original":29491200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19513344,"data_stored":81020596,"data_compressed":11156334,"data_compressed_allocated":14245888,"data_compressed_original":75771904,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":50333888,"data_compressed":7205527,"data_compressed_allocated":8712192,"data_compressed_original":46735360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8495104,"data_stored":42478411,"data_compressed":5618062,"data_compressed_allocated":6979584,"data_compressed_original":40984576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5763072,"data_stored":29361843,"data_compressed":4140996,"data_compressed_allocated":5042176,"data_compressed_original":28651520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":25700929,"data_compressed":3388457,"data_compressed_allocated":4177920,"data_compressed_original":23855104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285327360,"data_stored":312894956,"data_compressed":27257035,"data_compressed_allocated":30138368,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198365184,"data_stored":217026051,"data_compressed":18397674,"data_compressed_allocated":20283392,"data_compressed_original":44896256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220364800,"data_stored":244298213,"data_compressed":22109685,"data_compressed_allocated":24551424,"data_compressed_original":56172544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250929152,"data_stored":277753997,"data_compressed":28384565,"data_compressed_allocated":31244288,"data_compressed_original":66293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209920000,"data_stored":231648521,"data_compressed":19929697,"data_compressed_allocated":21929984,"data_compressed_original":49086464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239448064,"data_stored":261570437,"data_compressed":22618307,"data_compressed_allocated":25018368,"data_compressed_original":55078912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167698432,"data_stored":186276495,"data_compressed":18249245,"data_compressed_allocated":20205568,"data_compressed_original":44990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140419072,"data_stored":148778440,"data_compressed":7959868,"data_compressed_allocated":9097216,"data_compressed_original":22343680,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:22:10.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:22:10.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.800+0000 7f5dd4c00640 1 Processor -- start 2024-03-27T06:22:10.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.801+0000 7f5dd4c00640 1 -- start start 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5dd01549a0 0x7f5dd00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5dd00cb870 0x7f5dd006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd006be10 0x7f5dd015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd00cada0 con 0x7f5dd01549a0 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd015de10 con 0x7f5dd00cb870 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd015e0b0 con 0x7f5dd006be10 2024-03-27T06:22:10.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dcf400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5dd01549a0 0x7f5dd00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dc7e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5dd00cb870 0x7f5dd006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dcfe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd006be10 0x7f5dd015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dc7e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5dd00cb870 0x7f5dd006b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:48248/0 (socket says 172.21.15.132:48248) 2024-03-27T06:22:10.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.802+0000 7f5dc7e00640 1 -- 172.21.15.132:0/419020613 learned_addr learned my addr 172.21.15.132:0/419020613 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:10.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.803+0000 7f5dc7e00640 1 -- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd006be10 msgr2=0x7f5dd015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.803+0000 7f5dc7e00640 1 --2- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd006be10 0x7f5dd015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.803+0000 7f5dc7e00640 1 -- 172.21.15.132:0/419020613 >> v2:172.21.15.132:3300/0 conn(0x7f5dd01549a0 msgr2=0x7f5dd00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.803+0000 7f5dc7e00640 1 --2- 172.21.15.132:0/419020613 >> v2:172.21.15.132:3300/0 conn(0x7f5dd01549a0 0x7f5dd00cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.803+0000 7f5dc7e00640 1 -- 172.21.15.132:0/419020613 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd015e390 con 0x7f5dd00cb870 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.804+0000 7f5dc7e00640 1 --2- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd00cb870 0x7f5dd006b8d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00b2df0 tx=0x7f5db802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a8554e664fde194 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.804+0000 7f5dcea00640 1 -- 172.21.15.132:0/419020613 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8034070 con 0x7f5dd00cb870 2024-03-27T06:22:10.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.804+0000 7f5dcea00640 1 -- 172.21.15.132:0/419020613 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db8030860 con 0x7f5dd00cb870 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.804+0000 7f5dcea00640 1 -- 172.21.15.132:0/419020613 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8030c30 con 0x7f5dd00cb870 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 -- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd00cb870 msgr2=0x7f5dd006b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd00cb870 0x7f5dd006b8d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00b2df0 tx=0x7f5db802b9a0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 -- 172.21.15.132:0/419020613 shutdown_connections 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd006be10 0x7f5dd015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/419020613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd00cb870 0x7f5dd006b8d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/419020613 >> v2:172.21.15.132:3300/0 conn(0x7f5dd01549a0 0x7f5dd00cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 -- 172.21.15.132:0/419020613 >> 172.21.15.132:0/419020613 conn(0x7f5dd006d8d0 msgr2=0x7f5dd006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 -- 172.21.15.132:0/419020613 shutdown_connections 2024-03-27T06:22:10.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.805+0000 7f5dd4c00640 1 -- 172.21.15.132:0/419020613 wait complete. 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.807+0000 7f5dd4c00640 1 Processor -- start 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.807+0000 7f5dd4c00640 1 -- start start 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 0x7f5dd00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 0x7f5dd0150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd015ec30 con 0x7f5dd00cb870 2024-03-27T06:22:10.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd015e950 con 0x7f5dd006be10 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd015a820 con 0x7f5dd01549a0 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dc7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 0x7f5dd00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:48258/0 (socket says 172.21.15.132:48258) 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dc7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 0x7f5dd00b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53380/0 (socket says 172.21.15.132:53380) 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 -- 172.21.15.132:0/3155418289 learned_addr learned my addr 172.21.15.132:0/3155418289 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcfe00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 0x7f5dd0150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 -- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 msgr2=0x7f5dd0150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 0x7f5dd0150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 -- 172.21.15.132:0/3155418289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 msgr2=0x7f5dd00b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 0x7f5dd00b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcf400640 1 -- 172.21.15.132:0/3155418289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd014fef0 con 0x7f5dd006be10 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.808+0000 7f5dcfe00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 0x7f5dd0150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dcf400640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000e930 tx=0x7f5dc000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0017070 con 0x7f5dd006be10 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc000c9a0 con 0x7f5dd006be10 2024-03-27T06:22:10.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc000cd90 con 0x7f5dd006be10 2024-03-27T06:22:10.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd0153d80 con 0x7f5dd006be10 2024-03-27T06:22:10.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.809+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd0150160 con 0x7f5dd006be10 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.810+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd00686c0 con 0x7f5dd006be10 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.811+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5dc001a070 con 0x7f5dd006be10 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.812+0000 7f5dc7400640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 0x7f5dac06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.812+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f5dc008bca0 con 0x7f5dd006be10 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.812+0000 7f5dc7e00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 0x7f5dac06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:10.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.813+0000 7f5dc7e00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 0x7f5dac06e9f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5db8036000 tx=0x7f5db8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:10.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:10.820+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5dc0059ac0 con 0x7f5dd006be10 2024-03-27T06:22:11.001 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.012+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dd0158570 con 0x7f5dd006be10 2024-03-27T06:22:11.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.014+0000 7f5dc7400640 1 -- 172.21.15.132:0/3155418289 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17374 (secure 0 0 0) 0x7f5dc005ddc0 con 0x7f5dd006be10 2024-03-27T06:22:11.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:11.004 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":44,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:21:29.204148+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":40,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6839/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6837/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6838/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6837/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.134:6836/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6836/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6834/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6835/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.132:6837/690619582":"2024-03-28T06:15:19.112063+0000","172.21.15.132:6835/690619582":"2024-03-28T06:15:19.112063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:22:11.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 msgr2=0x7f5dac06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:11.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 0x7f5dac06e9f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5db8036000 tx=0x7f5db8008030 comp rx=0 tx=0).stop 2024-03-27T06:22:11.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 msgr2=0x7f5dd0152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:11.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000e930 tx=0x7f5dc000edf0 comp rx=0 tx=0).stop 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 shutdown_connections 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5dd01549a0 0x7f5dd0150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dd00cb870 0x7f5dd00b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5dac06c530 0x7f5dac06e9f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 --2- 172.21.15.132:0/3155418289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5dd006be10 0x7f5dd0152b80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.016+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 >> 172.21.15.132:0/3155418289 conn(0x7f5dd006d8d0 msgr2=0x7f5dd014a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.017+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 shutdown_connections 2024-03-27T06:22:11.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:11.017+0000 7f5dd4c00640 1 -- 172.21.15.132:0/3155418289 wait complete. 2024-03-27T06:22:12.024 INFO:tasks.ceph:Scrubbing osd.0 2024-03-27T06:22:12.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.255+0000 7f0002600640 1 Processor -- start 2024-03-27T06:22:12.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.255+0000 7f0002600640 1 -- start start 2024-03-27T06:22:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0002600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7efffc154a50 0x7efffc15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0002600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efffc154e30 0x7efffc06b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0002600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efffc15e480 con 0x7efffc15df40 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0002600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7efffc0cba40 con 0x7efffc154a50 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0002600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7efffc15e720 con 0x7efffc154e30 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7efffbe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efffc154e30 0x7efffc06b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0001c00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7efffc15df40 0x7efffc0cc200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60740/0 (socket says 172.21.15.132:60740) 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0001c00640 1 -- 172.21.15.132:0/3171646082 learned_addr learned my addr 172.21.15.132:0/3171646082 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.257+0000 7f0001200640 1 --2- 172.21.15.132:0/3171646082 >> v2:172.21.15.132:3300/0 conn(0x7efffc154a50 0x7efffc15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.258+0000 7efffbe00640 1 -- 172.21.15.132:0/3171646082 >> v2:172.21.15.132:3300/0 conn(0x7efffc154a50 msgr2=0x7efffc15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.258+0000 7efffbe00640 1 --2- 172.21.15.132:0/3171646082 >> v2:172.21.15.132:3300/0 conn(0x7efffc154a50 0x7efffc15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.258+0000 7efffbe00640 1 -- 172.21.15.132:0/3171646082 >> v1:172.21.15.134:6789/0 conn(0x7efffc15df40 legacy=0x7efffc0cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.258+0000 7efffbe00640 1 -- 172.21.15.132:0/3171646082 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efffc15ea00 con 0x7efffc154e30 2024-03-27T06:22:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.259+0000 7efffbe00640 1 --2- 172.21.15.132:0/3171646082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc06b8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efffc0b2df0 tx=0x7effe802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5ee4b5427a6f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.259+0000 7efffb400640 1 -- 172.21.15.132:0/3171646082 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8035020 con 0x7efffc154e30 2024-03-27T06:22:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.259+0000 7efffb400640 1 -- 172.21.15.132:0/3171646082 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8031830 con 0x7efffc154e30 2024-03-27T06:22:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.259+0000 7efffb400640 1 -- 172.21.15.132:0/3171646082 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8031c00 con 0x7efffc154e30 2024-03-27T06:22:12.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 -- 172.21.15.132:0/3171646082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 msgr2=0x7efffc06b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 --2- 172.21.15.132:0/3171646082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc06b8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efffc0b2df0 tx=0x7effe802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 -- 172.21.15.132:0/3171646082 shutdown_connections 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 --2- 172.21.15.132:0/3171646082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc06b8d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 --2- 172.21.15.132:0/3171646082 >> v2:172.21.15.132:3300/0 conn(0x7efffc154a50 0x7efffc15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 -- 172.21.15.132:0/3171646082 >> 172.21.15.132:0/3171646082 conn(0x7efffc06d910 msgr2=0x7efffc06dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 -- 172.21.15.132:0/3171646082 shutdown_connections 2024-03-27T06:22:12.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.260+0000 7f0002600640 1 -- 172.21.15.132:0/3171646082 wait complete. 2024-03-27T06:22:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.262+0000 7f0002600640 1 Processor -- start 2024-03-27T06:22:12.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.263+0000 7f0002600640 1 -- start start 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efffc154a50 0x7efffc0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc1f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efffc15df40 0x7efffc14a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efffc0cc910 con 0x7efffc154a50 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efffc0cbce0 con 0x7efffc154e30 2024-03-27T06:22:12.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7f0002600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7efffc0cca90 con 0x7efffc15df40 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.264+0000 7efffbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc1f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7efffbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc1f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37008/0 (socket says 172.21.15.132:37008) 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efffc15df40 0x7efffc14a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7efffbe00640 1 -- 172.21.15.132:0/3030977345 learned_addr learned my addr 172.21.15.132:0/3030977345 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001200640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efffc154a50 0x7efffc0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 -- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 msgr2=0x7efffc1f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc1f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 -- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efffc154a50 msgr2=0x7efffc0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efffc154a50 0x7efffc0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efffc1f4180 con 0x7efffc15df40 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7efffbe00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efffc154e30 0x7efffc1f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7f0001c00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efffc15df40 0x7efffc14a030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efff400ca50 tx=0x7efff400cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.265+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff4017070 con 0x7efffc15df40 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.266+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff400abb0 con 0x7efffc15df40 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.266+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff4012400 con 0x7efffc15df40 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.266+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efffc15d8c0 con 0x7efffc15df40 2024-03-27T06:22:12.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.266+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efffc0c5d50 con 0x7efffc15df40 2024-03-27T06:22:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.267+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efffc0ccfe0 con 0x7efffc15df40 2024-03-27T06:22:12.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.268+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7efff401a070 con 0x7efffc15df40 2024-03-27T06:22:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.269+0000 7efff8c00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7effdc06c560 0x7effdc06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.269+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7efff408b5b0 con 0x7efffc15df40 2024-03-27T06:22:12.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7f0001200640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7effdc06c560 0x7effdc06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7efff8c00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7effdc074ae0 0x7effdc076f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7effdc0775d0 con 0x7effdc074ae0 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7efff4092050 con 0x7efffc15df40 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7f0001200640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7effdc06c560 0x7effdc06ea20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efff0009aa0 tx=0x7efff0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.270+0000 7efffbe00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7effdc074ae0 0x7effdc076f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.271+0000 7efffbe00640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7effdc074ae0 0x7effdc076f00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.272+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== osd.0 v2:172.21.15.132:6800/1970178383 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7effdc0775d0 con 0x7effdc074ae0 2024-03-27T06:22:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.287+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 --> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efffc068730 con 0x7effdc074ae0 2024-03-27T06:22:12.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7efff8c00640 1 -- 172.21.15.132:0/3030977345 <== osd.0 v2:172.21.15.132:6800/1970178383 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efffc068730 con 0x7effdc074ae0 2024-03-27T06:22:12.278 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:12.278 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:12.278 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7effdc074ae0 msgr2=0x7effdc076f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.132:6800/1970178383,v1:172.21.15.132:6801/1970178383] conn(0x7effdc074ae0 0x7effdc076f00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7effdc06c560 msgr2=0x7effdc06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7effdc06c560 0x7effdc06ea20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efff0009aa0 tx=0x7efff0009190 comp rx=0 tx=0).stop 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efffc15df40 msgr2=0x7efffc14a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0002600640 1 --2- 172.21.15.132:0/3030977345 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7efffc15df40 0x7efffc14a030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efff400ca50 tx=0x7efff400cf10 comp rx=0 tx=0).stop 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.288+0000 7f0001c00640 1 -- 172.21.15.132:0/3030977345 reap_dead start 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.289+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 shutdown_connections 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.289+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 >> 172.21.15.132:0/3030977345 conn(0x7efffc06d910 msgr2=0x7efffc0ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.289+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 shutdown_connections 2024-03-27T06:22:12.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.289+0000 7f0002600640 1 -- 172.21.15.132:0/3030977345 wait complete. 2024-03-27T06:22:12.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-27T06:22:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.522+0000 7facff400640 1 Processor -- start 2024-03-27T06:22:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.522+0000 7facff400640 1 -- start start 2024-03-27T06:22:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.523+0000 7facff400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.523+0000 7facff400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7facf8072060 0x7facf80724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.523+0000 7facff400640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf80c7e00 con 0x7facf80c72c0 2024-03-27T06:22:12.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.523+0000 7facff400640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7facf80c9140 con 0x7facf8072060 2024-03-27T06:22:12.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facff400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7facf8067360 con 0x7facf8154d80 2024-03-27T06:22:12.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfd400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7facf8072060 0x7facf80724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:54942/0 (socket says 172.21.15.132:54942) 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 -- 172.21.15.132:0/4259952572 learned_addr learned my addr 172.21.15.132:0/4259952572 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 -- 172.21.15.132:0/4259952572 >> v2:172.21.15.134:3300/0 conn(0x7facf8072060 msgr2=0x7facf80724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 --2- 172.21.15.132:0/4259952572 >> v2:172.21.15.134:3300/0 conn(0x7facf8072060 0x7facf80724f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 -- 172.21.15.132:0/4259952572 >> v1:172.21.15.134:6789/0 conn(0x7facf80c72c0 legacy=0x7facf806b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facfca00640 1 -- 172.21.15.132:0/4259952572 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf80c80a0 con 0x7facf8154d80 2024-03-27T06:22:12.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.524+0000 7facf7400640 1 -- 172.21.15.132:0/4259952572 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134247867 0 0) 0x7facf80c7e00 con 0x7facf80c72c0 2024-03-27T06:22:12.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.525+0000 7facfca00640 1 --2- 172.21.15.132:0/4259952572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9e4113b70dd2d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facf7400640 1 -- 172.21.15.132:0/4259952572 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec034070 con 0x7facf8154d80 2024-03-27T06:22:12.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facf7400640 1 -- 172.21.15.132:0/4259952572 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec030840 con 0x7facf8154d80 2024-03-27T06:22:12.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facf7400640 1 -- 172.21.15.132:0/4259952572 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec030c10 con 0x7facf8154d80 2024-03-27T06:22:12.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 -- 172.21.15.132:0/4259952572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 msgr2=0x7facf815b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 --2- 172.21.15.132:0/4259952572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec02b980 comp rx=0 tx=0).stop 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 -- 172.21.15.132:0/4259952572 shutdown_connections 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 --2- 172.21.15.132:0/4259952572 >> v2:172.21.15.134:3300/0 conn(0x7facf8072060 0x7facf80724f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 --2- 172.21.15.132:0/4259952572 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8154d80 0x7facf815b3b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.526+0000 7facff400640 1 -- 172.21.15.132:0/4259952572 >> 172.21.15.132:0/4259952572 conn(0x7facf806d7c0 msgr2=0x7facf806dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.527+0000 7facff400640 1 -- 172.21.15.132:0/4259952572 shutdown_connections 2024-03-27T06:22:12.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.527+0000 7facff400640 1 -- 172.21.15.132:0/4259952572 wait complete. 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.528+0000 7facff400640 1 Processor -- start 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.528+0000 7facff400640 1 -- start start 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf80c72c0 0x7facf80c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facf8154d80 0x7facf8158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7facf80c8660 con 0x7facf80c72c0 2024-03-27T06:22:12.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7facf815b8f0 con 0x7facf8154d80 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facff400640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7facf80c8940 con 0x7facf8072060 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facfca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:54958/0 (socket says 172.21.15.132:54958) 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facf7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf80c72c0 0x7facf80c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facfd400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facf8154d80 0x7facf8158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.529+0000 7facfca00640 1 -- 172.21.15.132:0/1314121108 learned_addr learned my addr 172.21.15.132:0/1314121108 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 -- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facf8154d80 msgr2=0x7facf8158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facf8154d80 0x7facf8158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 -- 172.21.15.132:0/1314121108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf80c72c0 msgr2=0x7facf80c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf80c72c0 0x7facf80c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 -- 172.21.15.132:0/1314121108 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf815acf0 con 0x7facf8072060 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facfca00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7facec02b9c0 tx=0x7facec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec034040 con 0x7facf8072060 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec03ada0 con 0x7facf8072060 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.530+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf80c5e90 con 0x7facf8072060 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.531+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec044610 con 0x7facf8072060 2024-03-27T06:22:12.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.531+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facf80c6450 con 0x7facf8072060 2024-03-27T06:22:12.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.532+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf8068840 con 0x7facf8072060 2024-03-27T06:22:12.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.533+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7facec04b060 con 0x7facf8072060 2024-03-27T06:22:12.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.534+0000 7facf4c00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 0x7facd806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.534+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7facec0adb70 con 0x7facf8072060 2024-03-27T06:22:12.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.534+0000 7facf7e00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 0x7facd806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.535+0000 7facf7e00640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 0x7facd806e9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7facf815ef60 tx=0x7face4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.541+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7facec03f070 con 0x7facf8072060 2024-03-27T06:22:12.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.739+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7facf80c7910 con 0x7facd806c530 2024-03-27T06:22:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.740+0000 7facf4c00640 1 -- 172.21.15.132:0/1314121108 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7facf80c7910 con 0x7facd806c530 2024-03-27T06:22:12.730 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 0 to deep-scrub 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 msgr2=0x7facd806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 0x7facd806e9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7facf815ef60 tx=0x7face4009040 comp rx=0 tx=0).stop 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 msgr2=0x7facf81f05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7facec02b9c0 tx=0x7facec0046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 shutdown_connections 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7facd806c530 0x7facd806e9f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facf8154d80 0x7facf8158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf80c72c0 0x7facf80c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 --2- 172.21.15.132:0/1314121108 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7facf8072060 0x7facf81f05f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 >> 172.21.15.132:0/1314121108 conn(0x7facf806d7c0 msgr2=0x7facf8151dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 shutdown_connections 2024-03-27T06:22:12.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.743+0000 7facff400640 1 -- 172.21.15.132:0/1314121108 wait complete. 2024-03-27T06:22:12.751 INFO:tasks.ceph:Scrubbing osd.1 2024-03-27T06:22:12.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:12.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.970+0000 7ff2cbe00640 1 Processor -- start 2024-03-27T06:22:12.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.970+0000 7ff2cbe00640 1 -- start start 2024-03-27T06:22:12.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 0x7ff2cc158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cbe00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7ff2cc0cc590 0x7ff2cc0c1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cbe00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc15bca0 con 0x7ff2cc073300 2024-03-27T06:22:12.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cbe00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc0c0ec0 con 0x7ff2cc0cc590 2024-03-27T06:22:12.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cbe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc15bf60 con 0x7ff2cc0736e0 2024-03-27T06:22:12.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2caa00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff2cc073300 0x7ff2cc152060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60762/0 (socket says 172.21.15.132:60762) 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2caa00640 1 -- 172.21.15.132:0/4158946977 learned_addr learned my addr 172.21.15.132:0/4158946977 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2c2000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 0x7ff2cc158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.972+0000 7ff2cb400640 1 --2- 172.21.15.132:0/4158946977 >> v2:172.21.15.132:3300/0 conn(0x7ff2cc0cc590 0x7ff2cc0c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2ca000640 1 -- 172.21.15.132:0/4158946977 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031701994 0 0) 0x7ff2cc15bca0 con 0x7ff2cc073300 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2ca000640 1 -- 172.21.15.132:0/4158946977 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2b0003520 con 0x7ff2cc073300 2024-03-27T06:22:12.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2c2000640 1 -- 172.21.15.132:0/4158946977 >> v2:172.21.15.132:3300/0 conn(0x7ff2cc0cc590 msgr2=0x7ff2cc0c1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:12.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2c2000640 1 --2- 172.21.15.132:0/4158946977 >> v2:172.21.15.132:3300/0 conn(0x7ff2cc0cc590 0x7ff2cc0c1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2c2000640 1 -- 172.21.15.132:0/4158946977 >> v1:172.21.15.134:6789/0 conn(0x7ff2cc073300 legacy=0x7ff2cc152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.973+0000 7ff2c2000640 1 -- 172.21.15.132:0/4158946977 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc15c240 con 0x7ff2cc0736e0 2024-03-27T06:22:12.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.974+0000 7ff2c2000640 1 --2- 172.21.15.132:0/4158946977 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 0x7ff2cc158f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40097a0 tx=0x7ff2b402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea5398de837b4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.974+0000 7ff2ca000640 1 -- 172.21.15.132:0/4158946977 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4035020 con 0x7ff2cc0736e0 2024-03-27T06:22:12.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.974+0000 7ff2ca000640 1 -- 172.21.15.132:0/4158946977 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b4031810 con 0x7ff2cc0736e0 2024-03-27T06:22:12.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.974+0000 7ff2ca000640 1 -- 172.21.15.132:0/4158946977 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4031be0 con 0x7ff2cc0736e0 2024-03-27T06:22:12.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4158946977 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 msgr2=0x7ff2cc158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4158946977 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 0x7ff2cc158f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40097a0 tx=0x7ff2b402b8b0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4158946977 shutdown_connections 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4158946977 >> v2:172.21.15.132:3300/0 conn(0x7ff2cc0cc590 0x7ff2cc0c1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4158946977 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0736e0 0x7ff2cc158f70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4158946977 >> 172.21.15.132:0/4158946977 conn(0x7ff2cc0676a0 msgr2=0x7ff2cc06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4158946977 shutdown_connections 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.975+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4158946977 wait complete. 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.977+0000 7ff2cbe00640 1 Processor -- start 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.977+0000 7ff2cbe00640 1 -- start start 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2cc073300 0x7ff2cc1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2cc0736e0 0x7ff2cc1f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 0x7ff2cc14ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc0cbb20 con 0x7ff2cc0736e0 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc15c800 con 0x7ff2cc073300 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cbe00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc152770 con 0x7ff2cc0cc590 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 0x7ff2cc14ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2c2000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2cc0736e0 0x7ff2cc1f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2caa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2cc073300 0x7ff2cc1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 0x7ff2cc14ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:54980/0 (socket says 172.21.15.132:54980) 2024-03-27T06:22:12.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 -- 172.21.15.132:0/4040878695 learned_addr learned my addr 172.21.15.132:0/4040878695 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 -- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2cc073300 msgr2=0x7ff2cc1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2cc073300 0x7ff2cc1f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 -- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2cc0736e0 msgr2=0x7ff2cc1f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2cc0736e0 0x7ff2cc1f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2cb400640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc156220 con 0x7ff2cc0cc590 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.978+0000 7ff2c2000640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2cc0736e0 0x7ff2cc1f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:12.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2cb400640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 0x7ff2cc14ef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400d600 tx=0x7ff2c400dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40125f0 con 0x7ff2cc0cc590 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c4019070 con 0x7ff2cc0cc590 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40129d0 con 0x7ff2cc0cc590 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc156500 con 0x7ff2cc0cc590 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.979+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc156ac0 con 0x7ff2cc0cc590 2024-03-27T06:22:12.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.980+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff2cc068730 con 0x7ff2cc0cc590 2024-03-27T06:22:12.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.981+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7ff2c4012b70 con 0x7ff2cc0cc590 2024-03-27T06:22:12.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c3400640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff2a806c530 0x7ff2a806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7ff2c408bc60 con 0x7ff2cc0cc590 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2caa00640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff2a806c530 0x7ff2a806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c3400640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7ff2a8074ac0 0x7ff2a8076ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2c4016e70 con 0x7ff2a8074ac0 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff2c40902a0 con 0x7ff2cc0cc590 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2c2000640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7ff2a8074ac0 0x7ff2a8076ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.982+0000 7ff2caa00640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff2a806c530 0x7ff2a806e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005c00 tx=0x7ff2bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.983+0000 7ff2c2000640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7ff2a8074ac0 0x7ff2a8076ee0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:12.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.984+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== osd.1 v2:172.21.15.132:6813/3659088073 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2c4016e70 con 0x7ff2a8074ac0 2024-03-27T06:22:12.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.998+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 --> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff2cc0cb6b0 con 0x7ff2a8074ac0 2024-03-27T06:22:12.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.998+0000 7ff2c3400640 1 -- 172.21.15.132:0/4040878695 <== osd.1 v2:172.21.15.132:6813/3659088073 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff2cc0cb6b0 con 0x7ff2a8074ac0 2024-03-27T06:22:12.990 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:12.990 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:12.990 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:12.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7ff2a8074ac0 msgr2=0x7ff2a8076ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.132:6813/3659088073,v1:172.21.15.132:6814/3659088073] conn(0x7ff2a8074ac0 0x7ff2a8076ee0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff2a806c530 msgr2=0x7ff2a806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7ff2a806c530 0x7ff2a806e9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc005c00 tx=0x7ff2bc009040 comp rx=0 tx=0).stop 2024-03-27T06:22:12.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 msgr2=0x7ff2cc14ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:12.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 --2- 172.21.15.132:0/4040878695 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7ff2cc0cc590 0x7ff2cc14ef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400d600 tx=0x7ff2c400dac0 comp rx=0 tx=0).stop 2024-03-27T06:22:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cb400640 1 -- 172.21.15.132:0/4040878695 reap_dead start 2024-03-27T06:22:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 shutdown_connections 2024-03-27T06:22:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 >> 172.21.15.132:0/4040878695 conn(0x7ff2cc0676a0 msgr2=0x7ff2cc151ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 shutdown_connections 2024-03-27T06:22:12.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:12.999+0000 7ff2cbe00640 1 -- 172.21.15.132:0/4040878695 wait complete. 2024-03-27T06:22:13.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-27T06:22:13.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.219+0000 7eff3f000640 1 Processor -- start 2024-03-27T06:22:13.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.219+0000 7eff3f000640 1 -- start start 2024-03-27T06:22:13.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3f000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7eff380c8ab0 0x7eff38157d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3f000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7eff3806c630 0x7eff3806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3f000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff38071630 con 0x7eff38151800 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3f000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7eff380718d0 con 0x7eff380c8ab0 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3f000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7eff38071bb0 con 0x7eff3806c630 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3d200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7eff380c8ab0 0x7eff38157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3e600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7eff3806c630 0x7eff3806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3dc00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7eff38151800 0x7eff380c5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:45646/0 (socket says 172.21.15.132:45646) 2024-03-27T06:22:13.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.221+0000 7eff3dc00640 1 -- 172.21.15.132:0/949116987 learned_addr learned my addr 172.21.15.132:0/949116987 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:13.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.222+0000 7eff3e600640 1 -- 172.21.15.132:0/949116987 >> v2:172.21.15.132:3300/0 conn(0x7eff380c8ab0 msgr2=0x7eff38157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.222+0000 7eff3e600640 1 --2- 172.21.15.132:0/949116987 >> v2:172.21.15.132:3300/0 conn(0x7eff380c8ab0 0x7eff38157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.222+0000 7eff3e600640 1 -- 172.21.15.132:0/949116987 >> v1:172.21.15.132:6789/0 conn(0x7eff38151800 legacy=0x7eff380c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.222+0000 7eff3e600640 1 -- 172.21.15.132:0/949116987 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff380c5140 con 0x7eff3806c630 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.223+0000 7eff3e600640 1 --2- 172.21.15.132:0/949116987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff3806c630 0x7eff3806ca60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff380c4de0 tx=0x7eff3002be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac45946ee71472fc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.223+0000 7eff37e00640 1 -- 172.21.15.132:0/949116987 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30039070 con 0x7eff3806c630 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.223+0000 7eff37e00640 1 -- 172.21.15.132:0/949116987 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3002cbe0 con 0x7eff3806c630 2024-03-27T06:22:13.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.223+0000 7eff37e00640 1 -- 172.21.15.132:0/949116987 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30034440 con 0x7eff3806c630 2024-03-27T06:22:13.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 -- 172.21.15.132:0/949116987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff3806c630 msgr2=0x7eff3806ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 --2- 172.21.15.132:0/949116987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff3806c630 0x7eff3806ca60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff380c4de0 tx=0x7eff3002be30 comp rx=0 tx=0).stop 2024-03-27T06:22:13.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 -- 172.21.15.132:0/949116987 shutdown_connections 2024-03-27T06:22:13.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 --2- 172.21.15.132:0/949116987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff3806c630 0x7eff3806ca60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 --2- 172.21.15.132:0/949116987 >> v2:172.21.15.132:3300/0 conn(0x7eff380c8ab0 0x7eff38157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 -- 172.21.15.132:0/949116987 >> 172.21.15.132:0/949116987 conn(0x7eff380c9000 msgr2=0x7eff380c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:13.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 -- 172.21.15.132:0/949116987 shutdown_connections 2024-03-27T06:22:13.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.224+0000 7eff3f000640 1 -- 172.21.15.132:0/949116987 wait complete. 2024-03-27T06:22:13.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.225+0000 7eff3f000640 1 Processor -- start 2024-03-27T06:22:13.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.226+0000 7eff3f000640 1 -- start start 2024-03-27T06:22:13.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff3806c630 0x7eff3814fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff38151800 0x7eff38146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff380c59e0 con 0x7eff3806c630 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7eff380c5700 con 0x7eff38151800 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3f000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7eff380c6920 con 0x7eff380c8ab0 2024-03-27T06:22:13.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff3806c630 0x7eff3814fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:54982/0 (socket says 172.21.15.132:54982) 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 -- 172.21.15.132:0/2591714139 learned_addr learned my addr 172.21.15.132:0/2591714139 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3e600640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff38151800 0x7eff38146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 -- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff38151800 msgr2=0x7eff38146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff38151800 0x7eff38146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 -- 172.21.15.132:0/2591714139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff3806c630 msgr2=0x7eff3814fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.227+0000 7eff3d200640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff3806c630 0x7eff3814fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff3d200640 1 -- 172.21.15.132:0/2591714139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff381f10a0 con 0x7eff380c8ab0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff3d200640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff2800eb40 tx=0x7eff2800d350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28010440 con 0x7eff380c8ab0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff280105e0 con 0x7eff380c8ab0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff381f1340 con 0x7eff380c8ab0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff280109d0 con 0x7eff380c8ab0 2024-03-27T06:22:13.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.228+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff381f1900 con 0x7eff380c8ab0 2024-03-27T06:22:13.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.230+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3814afe0 con 0x7eff380c8ab0 2024-03-27T06:22:13.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.230+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7eff28010b70 con 0x7eff380c8ab0 2024-03-27T06:22:13.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.231+0000 7eff35600640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 0x7eff1c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.231+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7eff2808c190 con 0x7eff380c8ab0 2024-03-27T06:22:13.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.231+0000 7eff3dc00640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 0x7eff1c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.232+0000 7eff3dc00640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 0x7eff1c06e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff2c007fd0 tx=0x7eff2c007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.236+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7eff28014030 con 0x7eff380c8ab0 2024-03-27T06:22:13.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.434+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7eff380c8890 con 0x7eff1c06c530 2024-03-27T06:22:13.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.435+0000 7eff35600640 1 -- 172.21.15.132:0/2591714139 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7eff380c8890 con 0x7eff1c06c530 2024-03-27T06:22:13.426 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 1 to deep-scrub 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 msgr2=0x7eff1c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 0x7eff1c06e9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff2c007fd0 tx=0x7eff2c007e80 comp rx=0 tx=0).stop 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 msgr2=0x7eff38150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff2800eb40 tx=0x7eff2800d350 comp rx=0 tx=0).stop 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 shutdown_connections 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7eff1c06c530 0x7eff1c06e9f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff38151800 0x7eff38146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7eff380c8ab0 0x7eff38150380 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 --2- 172.21.15.132:0/2591714139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff3806c630 0x7eff3814fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 >> 172.21.15.132:0/2591714139 conn(0x7eff380c9000 msgr2=0x7eff380c2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:13.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 shutdown_connections 2024-03-27T06:22:13.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.438+0000 7eff3f000640 1 -- 172.21.15.132:0/2591714139 wait complete. 2024-03-27T06:22:13.449 INFO:tasks.ceph:Scrubbing osd.2 2024-03-27T06:22:13.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:13.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.691+0000 7fa554a00640 1 Processor -- start 2024-03-27T06:22:13.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.691+0000 7fa554a00640 1 -- start start 2024-03-27T06:22:13.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.692+0000 7fa554a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 0x7fa55015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa554a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa554a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55015e480 con 0x7fa550154a50 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa554a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fa5500cba40 con 0x7fa550154e30 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa554a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa55015e720 con 0x7fa55015df40 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54ea00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 0x7fa55015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:54992/0 (socket says 172.21.15.132:54992) 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 -- 172.21.15.132:0/4277376898 learned_addr learned my addr 172.21.15.132:0/4277376898 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:13.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54ea00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 0x7fa55015b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37072/0 (socket says 172.21.15.132:37072) 2024-03-27T06:22:13.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 -- 172.21.15.132:0/4277376898 >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 msgr2=0x7fa55015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:13.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 --2- 172.21.15.132:0/4277376898 >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 0x7fa55015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 -- 172.21.15.132:0/4277376898 >> v1:172.21.15.132:6789/0 conn(0x7fa550154a50 legacy=0x7fa55006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.693+0000 7fa54fe00640 1 -- 172.21.15.132:0/4277376898 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55015ea00 con 0x7fa55015df40 2024-03-27T06:22:13.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.694+0000 7fa54fe00640 1 --2- 172.21.15.132:0/4277376898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e50 tx=0x7fa54802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2634e43c9c15997a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa54e000640 1 -- 172.21.15.132:0/4277376898 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54802cb00 con 0x7fa55015df40 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa54e000640 1 -- 172.21.15.132:0/4277376898 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa54802cca0 con 0x7fa55015df40 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa54e000640 1 -- 172.21.15.132:0/4277376898 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548034610 con 0x7fa55015df40 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 -- 172.21.15.132:0/4277376898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 msgr2=0x7fa5500cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 --2- 172.21.15.132:0/4277376898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e50 tx=0x7fa54802bf60 comp rx=0 tx=0).stop 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 -- 172.21.15.132:0/4277376898 shutdown_connections 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 --2- 172.21.15.132:0/4277376898 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5500cc200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 --2- 172.21.15.132:0/4277376898 >> v2:172.21.15.134:3300/0 conn(0x7fa550154e30 0x7fa55015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.695+0000 7fa554a00640 1 -- 172.21.15.132:0/4277376898 >> 172.21.15.132:0/4277376898 conn(0x7fa55006d910 msgr2=0x7fa55006dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:13.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.696+0000 7fa554a00640 1 -- 172.21.15.132:0/4277376898 shutdown_connections 2024-03-27T06:22:13.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.696+0000 7fa554a00640 1 -- 172.21.15.132:0/4277376898 wait complete. 2024-03-27T06:22:13.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.698+0000 7fa554a00640 1 Processor -- start 2024-03-27T06:22:13.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.698+0000 7fa554a00640 1 -- start start 2024-03-27T06:22:13.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa550154a50 0x7fa55014e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 0x7fa5501f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5501f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa55006bfe0 con 0x7fa550154a50 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5500cbce0 con 0x7fa550154e30 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.699+0000 7fa554a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fa55006c160 con 0x7fa55015df40 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 0x7fa5501f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 0x7fa5501f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37080/0 (socket says 172.21.15.132:37080) 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 -- 172.21.15.132:0/2549116744 learned_addr learned my addr 172.21.15.132:0/2549116744 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:13.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54f400640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa550154a50 0x7fa55014e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54fe00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5501f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 -- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 msgr2=0x7fa5501f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fa55015df40 0x7fa5501f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 -- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa550154a50 msgr2=0x7fa55014e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa550154a50 0x7fa55014e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa550147e00 con 0x7fa550154e30 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.700+0000 7fa54ea00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 0x7fa5501f3810 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e9b0 tx=0x7fa53c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.701+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c017070 con 0x7fa550154e30 2024-03-27T06:22:13.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.701+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53c00c9a0 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.701+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c00cd90 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.701+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5501f4140 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.701+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501f1f50 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.702+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa550068730 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.703+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fa53c01a070 con 0x7fa550154e30 2024-03-27T06:22:13.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa547400640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa53006c530 0x7fa53006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fa53c08b9f0 con 0x7fa550154e30 2024-03-27T06:22:13.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa54f400640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa53006c530 0x7fa53006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa547400640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7fa530074ab0 0x7fa530076ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5300775a0 con 0x7fa530074ab0 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa53c08bda0 con 0x7fa550154e30 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.704+0000 7fa54fe00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7fa530074ab0 0x7fa530076ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.705+0000 7fa54f400640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa53006c530 0x7fa53006e9f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa540000f60 tx=0x7fa540009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.705+0000 7fa54fe00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7fa530074ab0 0x7fa530076ed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.706+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== osd.2 v2:172.21.15.132:6823/1881883030 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5300775a0 con 0x7fa530074ab0 2024-03-27T06:22:13.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.720+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 --> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa5500cb0a0 con 0x7fa530074ab0 2024-03-27T06:22:13.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa547400640 1 -- 172.21.15.132:0/2549116744 <== osd.2 v2:172.21.15.132:6823/1881883030 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa5500cb0a0 con 0x7fa530074ab0 2024-03-27T06:22:13.712 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:13.712 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:13.712 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7fa530074ab0 msgr2=0x7fa530076ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.132:6823/1881883030,v1:172.21.15.132:6825/1881883030] conn(0x7fa530074ab0 0x7fa530076ed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa53006c530 msgr2=0x7fa53006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fa53006c530 0x7fa53006e9f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa540000f60 tx=0x7fa540009040 comp rx=0 tx=0).stop 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 msgr2=0x7fa5501f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 --2- 172.21.15.132:0/2549116744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa550154e30 0x7fa5501f3810 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e9b0 tx=0x7fa53c00ee70 comp rx=0 tx=0).stop 2024-03-27T06:22:13.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa54fe00640 1 -- 172.21.15.132:0/2549116744 reap_dead start 2024-03-27T06:22:13.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 shutdown_connections 2024-03-27T06:22:13.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 >> 172.21.15.132:0/2549116744 conn(0x7fa55006d910 msgr2=0x7fa5500c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:13.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 shutdown_connections 2024-03-27T06:22:13.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.721+0000 7fa554a00640 1 -- 172.21.15.132:0/2549116744 wait complete. 2024-03-27T06:22:13.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-27T06:22:13.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.940+0000 7f2a31a00640 1 Processor -- start 2024-03-27T06:22:13.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.940+0000 7f2a31a00640 1 -- start start 2024-03-27T06:22:13.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a31a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f2a2c1516f0 0x7f2a2c0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a31a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2a2c157c80 0x7f2a2c0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a31a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2c06b520 con 0x7f2a2c151ad0 2024-03-27T06:22:13.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a31a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2c072be0 con 0x7f2a2c1516f0 2024-03-27T06:22:13.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a31a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2c06b7c0 con 0x7f2a2c157c80 2024-03-27T06:22:13.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a22000640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f2a2c151ad0 0x7f2a2c157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:45666/0 (socket says 172.21.15.132:45666) 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.942+0000 7f2a22000640 1 -- 172.21.15.132:0/483898561 learned_addr learned my addr 172.21.15.132:0/483898561 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a2aa00640 1 --2- 172.21.15.132:0/483898561 >> v2:172.21.15.132:3300/0 conn(0x7f2a2c1516f0 0x7f2a2c0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.941+0000 7f2a2b400640 1 --2- 172.21.15.132:0/483898561 >> v2:172.21.15.134:3300/0 conn(0x7f2a2c157c80 0x7f2a2c0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.942+0000 7f2a2b400640 1 -- 172.21.15.132:0/483898561 >> v2:172.21.15.132:3300/0 conn(0x7f2a2c1516f0 msgr2=0x7f2a2c0c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.942+0000 7f2a2b400640 1 --2- 172.21.15.132:0/483898561 >> v2:172.21.15.132:3300/0 conn(0x7f2a2c1516f0 0x7f2a2c0c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.942+0000 7f2a2b400640 1 -- 172.21.15.132:0/483898561 >> v1:172.21.15.132:6789/0 conn(0x7f2a2c151ad0 legacy=0x7f2a2c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.942+0000 7f2a2b400640 1 -- 172.21.15.132:0/483898561 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c06baa0 con 0x7f2a2c157c80 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.943+0000 7f2a2b400640 1 --2- 172.21.15.132:0/483898561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c157c80 0x7f2a2c0c8360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0c7dd0 tx=0x7f2a2402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb081f09184f855 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.943+0000 7f2a2a000640 1 -- 172.21.15.132:0/483898561 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24034070 con 0x7f2a2c157c80 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.943+0000 7f2a2a000640 1 -- 172.21.15.132:0/483898561 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a24030610 con 0x7f2a2c157c80 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.943+0000 7f2a2a000640 1 -- 172.21.15.132:0/483898561 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240309e0 con 0x7f2a2c157c80 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 -- 172.21.15.132:0/483898561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c157c80 msgr2=0x7f2a2c0c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 --2- 172.21.15.132:0/483898561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c157c80 0x7f2a2c0c8360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0c7dd0 tx=0x7f2a2402b960 comp rx=0 tx=0).stop 2024-03-27T06:22:13.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 -- 172.21.15.132:0/483898561 shutdown_connections 2024-03-27T06:22:13.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 --2- 172.21.15.132:0/483898561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c157c80 0x7f2a2c0c8360 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 --2- 172.21.15.132:0/483898561 >> v2:172.21.15.132:3300/0 conn(0x7f2a2c1516f0 0x7f2a2c0c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 -- 172.21.15.132:0/483898561 >> 172.21.15.132:0/483898561 conn(0x7f2a2c0cc060 msgr2=0x7f2a2c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:13.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 -- 172.21.15.132:0/483898561 shutdown_connections 2024-03-27T06:22:13.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.944+0000 7f2a31a00640 1 -- 172.21.15.132:0/483898561 wait complete. 2024-03-27T06:22:13.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.946+0000 7f2a31a00640 1 Processor -- start 2024-03-27T06:22:13.937 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.946+0000 7f2a31a00640 1 -- start start 2024-03-27T06:22:13.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a2c151ad0 0x7f2a2c1df420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2c157c80 0x7f2a2c1df960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c06bd80 con 0x7f2a2c157c80 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c072870 con 0x7f2a2c1516f0 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a31a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2c0c1080 con 0x7f2a2c151ad0 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37094/0 (socket says 172.21.15.132:37094) 2024-03-27T06:22:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 -- 172.21.15.132:0/1251783892 learned_addr learned my addr 172.21.15.132:0/1251783892 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2b400640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2c157c80 0x7f2a2c1df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a22000640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a2c151ad0 0x7f2a2c1df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 -- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a2c151ad0 msgr2=0x7f2a2c1df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a2c151ad0 0x7f2a2c1df420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 -- 172.21.15.132:0/1251783892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2c157c80 msgr2=0x7f2a2c1df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2c157c80 0x7f2a2c1df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.947+0000 7f2a2aa00640 1 -- 172.21.15.132:0/1251783892 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c1efaf0 con 0x7f2a2c1516f0 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a2aa00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00efc0 tx=0x7f2a1c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c018020 con 0x7f2a2c1516f0 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a1c014480 con 0x7f2a2c1516f0 2024-03-27T06:22:13.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2c1de940 con 0x7f2a2c1516f0 2024-03-27T06:22:13.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c014850 con 0x7f2a2c1516f0 2024-03-27T06:22:13.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.948+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2c1def00 con 0x7f2a2c1516f0 2024-03-27T06:22:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.949+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2c068840 con 0x7f2a2c1516f0 2024-03-27T06:22:13.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.950+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2a1c005d10 con 0x7f2a2c1516f0 2024-03-27T06:22:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.951+0000 7f2a23400640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 0x7f2a0806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.951+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f2a1c08c300 con 0x7f2a2c1516f0 2024-03-27T06:22:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.951+0000 7f2a22000640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 0x7f2a0806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.952+0000 7f2a22000640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 0x7f2a0806ea20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0c1530 tx=0x7f2a14007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:13.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:13.956+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2a1c022070 con 0x7f2a2c1516f0 2024-03-27T06:22:14.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.171+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f2a2c1de0c0 con 0x7f2a0806c560 2024-03-27T06:22:14.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.172+0000 7f2a23400640 1 -- 172.21.15.132:0/1251783892 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2a14001460 con 0x7f2a0806c560 2024-03-27T06:22:14.163 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 2 to deep-scrub 2024-03-27T06:22:14.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 msgr2=0x7f2a0806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 0x7f2a0806ea20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0c1530 tx=0x7f2a14007e80 comp rx=0 tx=0).stop 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 msgr2=0x7f2a2c1ef1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00efc0 tx=0x7f2a1c00c490 comp rx=0 tx=0).stop 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 shutdown_connections 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2c157c80 0x7f2a2c1df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2a2c151ad0 0x7f2a2c1df420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2a0806c560 0x7f2a0806ea20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 --2- 172.21.15.132:0/1251783892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c1516f0 0x7f2a2c1ef1c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 >> 172.21.15.132:0/1251783892 conn(0x7f2a2c0cc060 msgr2=0x7f2a2c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:14.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 shutdown_connections 2024-03-27T06:22:14.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.176+0000 7f2a31a00640 1 -- 172.21.15.132:0/1251783892 wait complete. 2024-03-27T06:22:14.186 INFO:tasks.ceph:Scrubbing osd.3 2024-03-27T06:22:14.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:14.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.434+0000 7fe666600640 1 Processor -- start 2024-03-27T06:22:14.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.434+0000 7fe666600640 1 -- start start 2024-03-27T06:22:14.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe666600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe660154a50 0x7fe66015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe666600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 0x7fe6600cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe666600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe66015e480 con 0x7fe660154e30 2024-03-27T06:22:14.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe666600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fe6600cba40 con 0x7fe660154a50 2024-03-27T06:22:14.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe666600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe66015e720 con 0x7fe66015df40 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 0x7fe6600cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe65fe00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe660154e30 0x7fe66006b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60778/0 (socket says 172.21.15.132:60778) 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe65fe00640 1 -- 172.21.15.132:0/1946605184 learned_addr learned my addr 172.21.15.132:0/1946605184 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665200640 1 --2- 172.21.15.132:0/1946605184 >> v2:172.21.15.132:3300/0 conn(0x7fe660154a50 0x7fe66015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665c00640 1 -- 172.21.15.132:0/1946605184 >> v2:172.21.15.132:3300/0 conn(0x7fe660154a50 msgr2=0x7fe66015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665c00640 1 --2- 172.21.15.132:0/1946605184 >> v2:172.21.15.132:3300/0 conn(0x7fe660154a50 0x7fe66015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665c00640 1 -- 172.21.15.132:0/1946605184 >> v1:172.21.15.134:6789/0 conn(0x7fe660154e30 legacy=0x7fe66006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe65f400640 1 -- 172.21.15.132:0/1946605184 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840966697 0 0) 0x7fe66015e480 con 0x7fe660154e30 2024-03-27T06:22:14.429 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.436+0000 7fe665c00640 1 -- 172.21.15.132:0/1946605184 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66015ea00 con 0x7fe66015df40 2024-03-27T06:22:14.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.438+0000 7fe665c00640 1 --2- 172.21.15.132:0/1946605184 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 0x7fe6600cc200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe658009d90 tx=0x7fe65802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1535c8a4a7388475 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.438+0000 7fe65f400640 1 -- 172.21.15.132:0/1946605184 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658039070 con 0x7fe66015df40 2024-03-27T06:22:14.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.438+0000 7fe65f400640 1 -- 172.21.15.132:0/1946605184 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe65802cc50 con 0x7fe66015df40 2024-03-27T06:22:14.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.438+0000 7fe65f400640 1 -- 172.21.15.132:0/1946605184 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658034440 con 0x7fe66015df40 2024-03-27T06:22:14.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 -- 172.21.15.132:0/1946605184 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 msgr2=0x7fe6600cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 --2- 172.21.15.132:0/1946605184 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 0x7fe6600cc200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe658009d90 tx=0x7fe65802bea0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 -- 172.21.15.132:0/1946605184 shutdown_connections 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 --2- 172.21.15.132:0/1946605184 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe66015df40 0x7fe6600cc200 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 --2- 172.21.15.132:0/1946605184 >> v2:172.21.15.132:3300/0 conn(0x7fe660154a50 0x7fe66015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 -- 172.21.15.132:0/1946605184 >> 172.21.15.132:0/1946605184 conn(0x7fe66006d910 msgr2=0x7fe66006dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 -- 172.21.15.132:0/1946605184 shutdown_connections 2024-03-27T06:22:14.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.439+0000 7fe666600640 1 -- 172.21.15.132:0/1946605184 wait complete. 2024-03-27T06:22:14.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.442+0000 7fe666600640 1 Processor -- start 2024-03-27T06:22:14.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.442+0000 7fe666600640 1 -- start start 2024-03-27T06:22:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe660154a50 0x7fe6600cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe660154e30 0x7fe6600cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 0x7fe6600d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe66006bfe0 con 0x7fe660154a50 2024-03-27T06:22:14.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe6600cbce0 con 0x7fe66015df40 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.443+0000 7fe666600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe66006c160 con 0x7fe660154e30 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 0x7fe6600d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe65fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe660154e30 0x7fe6600cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe660154a50 0x7fe6600cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 0x7fe6600d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37110/0 (socket says 172.21.15.132:37110) 2024-03-27T06:22:14.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 -- 172.21.15.132:0/4147184772 learned_addr learned my addr 172.21.15.132:0/4147184772 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:14.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 -- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe660154e30 msgr2=0x7fe6600cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe660154e30 0x7fe6600cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 -- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe660154a50 msgr2=0x7fe6600cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe660154a50 0x7fe6600cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6600d0e50 con 0x7fe66015df40 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.444+0000 7fe665c00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 0x7fe6600d0200 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe658009d90 tx=0x7fe658004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.445+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658039050 con 0x7fe66015df40 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.445+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe658045070 con 0x7fe66015df40 2024-03-27T06:22:14.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.445+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6601f0790 con 0x7fe66015df40 2024-03-27T06:22:14.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.445+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658009450 con 0x7fe66015df40 2024-03-27T06:22:14.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.445+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6600ce4d0 con 0x7fe66015df40 2024-03-27T06:22:14.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.446+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe6600c67a0 con 0x7fe66015df40 2024-03-27T06:22:14.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.447+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe65803e610 con 0x7fe66015df40 2024-03-27T06:22:14.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.448+0000 7fe65cc00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe64006c530 0x7fe64006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.448+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fe6580ae240 con 0x7fe66015df40 2024-03-27T06:22:14.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.448+0000 7fe665200640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe64006c530 0x7fe64006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.449+0000 7fe65cc00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fe640074ac0 0x7fe640076ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.449+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe65802cc50 con 0x7fe640074ac0 2024-03-27T06:22:14.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.449+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe6580b5050 con 0x7fe66015df40 2024-03-27T06:22:14.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.449+0000 7fe65fe00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fe640074ac0 0x7fe640076ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.449+0000 7fe665200640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe64006c530 0x7fe64006e9f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe654009aa0 tx=0x7fe654009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.450+0000 7fe65fe00640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fe640074ac0 0x7fe640076ee0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.450+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== osd.3 v2:172.21.15.132:6808/2165878941 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe65802cc50 con 0x7fe640074ac0 2024-03-27T06:22:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 --> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe660068730 con 0x7fe640074ac0 2024-03-27T06:22:14.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe65cc00640 1 -- 172.21.15.132:0/4147184772 <== osd.3 v2:172.21.15.132:6808/2165878941 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe660068730 con 0x7fe640074ac0 2024-03-27T06:22:14.456 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:14.456 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:14.456 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:14.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fe640074ac0 msgr2=0x7fe640076ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.132:6808/2165878941,v1:172.21.15.132:6809/2165878941] conn(0x7fe640074ac0 0x7fe640076ee0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe64006c530 msgr2=0x7fe64006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe64006c530 0x7fe64006e9f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe654009aa0 tx=0x7fe654009190 comp rx=0 tx=0).stop 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 msgr2=0x7fe6600d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.465+0000 7fe666600640 1 --2- 172.21.15.132:0/4147184772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe66015df40 0x7fe6600d0200 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe658009d90 tx=0x7fe658004a40 comp rx=0 tx=0).stop 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.466+0000 7fe665c00640 1 -- 172.21.15.132:0/4147184772 reap_dead start 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.466+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 shutdown_connections 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.466+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 >> 172.21.15.132:0/4147184772 conn(0x7fe66006d910 msgr2=0x7fe66006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.466+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 shutdown_connections 2024-03-27T06:22:14.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.466+0000 7fe666600640 1 -- 172.21.15.132:0/4147184772 wait complete. 2024-03-27T06:22:14.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-27T06:22:14.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.676+0000 7fc1edc00640 1 Processor -- start 2024-03-27T06:22:14.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.676+0000 7fc1edc00640 1 -- start start 2024-03-27T06:22:14.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.677+0000 7fc1edc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc1e8073610 0x7fc1e81579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1edc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fc1e80c7a00 0x7fc1e8157f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1edc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e80c9130 con 0x7fc1e81517b0 2024-03-27T06:22:14.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1edc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e80c8380 con 0x7fc1e80c7a00 2024-03-27T06:22:14.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1edc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e80c85b0 con 0x7fc1e8073610 2024-03-27T06:22:14.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc1e8073610 0x7fc1e81579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6a00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc1e81517b0 0x7fc1e80c9a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60790/0 (socket says 172.21.15.132:60790) 2024-03-27T06:22:14.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6a00640 1 -- 172.21.15.132:0/1088150397 learned_addr learned my addr 172.21.15.132:0/1088150397 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:14.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e7400640 1 --2- 172.21.15.132:0/1088150397 >> v2:172.21.15.132:3300/0 conn(0x7fc1e80c7a00 0x7fc1e8157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6000640 1 -- 172.21.15.132:0/1088150397 >> v2:172.21.15.132:3300/0 conn(0x7fc1e80c7a00 msgr2=0x7fc1e8157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:14.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6000640 1 --2- 172.21.15.132:0/1088150397 >> v2:172.21.15.132:3300/0 conn(0x7fc1e80c7a00 0x7fc1e8157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6000640 1 -- 172.21.15.132:0/1088150397 >> v1:172.21.15.134:6789/0 conn(0x7fc1e81517b0 legacy=0x7fc1e80c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.678+0000 7fc1e6000640 1 -- 172.21.15.132:0/1088150397 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80c8890 con 0x7fc1e8073610 2024-03-27T06:22:14.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.679+0000 7fc1e6000640 1 --2- 172.21.15.132:0/1088150397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e81579e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80caaf0 tx=0x7fc1d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c360aa94f163d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.680+0000 7fc1e5600640 1 -- 172.21.15.132:0/1088150397 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4035020 con 0x7fc1e8073610 2024-03-27T06:22:14.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.680+0000 7fc1e5600640 1 -- 172.21.15.132:0/1088150397 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d4031830 con 0x7fc1e8073610 2024-03-27T06:22:14.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.680+0000 7fc1e5600640 1 -- 172.21.15.132:0/1088150397 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4031c00 con 0x7fc1e8073610 2024-03-27T06:22:14.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.680+0000 7fc1edc00640 1 -- 172.21.15.132:0/1088150397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 msgr2=0x7fc1e81579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.680+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1088150397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e81579e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80caaf0 tx=0x7fc1d402b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 -- 172.21.15.132:0/1088150397 shutdown_connections 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1088150397 >> v2:172.21.15.132:3300/0 conn(0x7fc1e80c7a00 0x7fc1e8157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1088150397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e81579e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 -- 172.21.15.132:0/1088150397 >> 172.21.15.132:0/1088150397 conn(0x7fc1e806fc20 msgr2=0x7fc1e8070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 -- 172.21.15.132:0/1088150397 shutdown_connections 2024-03-27T06:22:14.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.681+0000 7fc1edc00640 1 -- 172.21.15.132:0/1088150397 wait complete. 2024-03-27T06:22:14.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.683+0000 7fc1edc00640 1 Processor -- start 2024-03-27T06:22:14.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.683+0000 7fc1edc00640 1 -- start start 2024-03-27T06:22:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc1e80c7a00 0x7fc1e81f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 0x7fc1e8142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e8072280 con 0x7fc1e81517b0 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80c8e50 con 0x7fc1e8073610 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1edc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e8151b90 con 0x7fc1e80c7a00 2024-03-27T06:22:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1e6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1e6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37116/0 (socket says 172.21.15.132:37116) 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1e6a00640 1 -- 172.21.15.132:0/1167826750 learned_addr learned my addr 172.21.15.132:0/1167826750 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1e6000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc1e80c7a00 0x7fc1e81f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.685+0000 7fc1e7400640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 0x7fc1e8142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 -- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc1e80c7a00 msgr2=0x7fc1e81f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc1e80c7a00 0x7fc1e81f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 -- 172.21.15.132:0/1167826750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 msgr2=0x7fc1e8142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 0x7fc1e8142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 -- 172.21.15.132:0/1167826750 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e81540e0 con 0x7fc1e8073610 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e7400640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 0x7fc1e8142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1e6a00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c3c0 tx=0x7fc1d800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8017070 con 0x7fc1e8073610 2024-03-27T06:22:14.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d8005ce0 con 0x7fc1e8073610 2024-03-27T06:22:14.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e81448d0 con 0x7fc1e8073610 2024-03-27T06:22:14.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80072f0 con 0x7fc1e8073610 2024-03-27T06:22:14.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.686+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e8154660 con 0x7fc1e8073610 2024-03-27T06:22:14.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.688+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e8068840 con 0x7fc1e8073610 2024-03-27T06:22:14.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.688+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fc1d801a070 con 0x7fc1e8073610 2024-03-27T06:22:14.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.689+0000 7fc1dea00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 0x7fc1c806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:14.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.689+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fc1d808aae0 con 0x7fc1e8073610 2024-03-27T06:22:14.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.689+0000 7fc1e6000640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 0x7fc1c806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:14.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.690+0000 7fc1e6000640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 0x7fc1c806e9f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:14.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.698+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fc1e80c9130 con 0x7fc1e8073610 2024-03-27T06:22:14.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.921+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc1e8157b00 con 0x7fc1c806c530 2024-03-27T06:22:14.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.922+0000 7fc1dea00640 1 -- 172.21.15.132:0/1167826750 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc1e8157b00 con 0x7fc1c806c530 2024-03-27T06:22:14.913 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 3 to deep-scrub 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 msgr2=0x7fc1c806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 0x7fc1c806e9f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4036000 tx=0x7fc1d4008030 comp rx=0 tx=0).stop 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 msgr2=0x7fc1e8148f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c3c0 tx=0x7fc1d800c880 comp rx=0 tx=0).stop 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 shutdown_connections 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1e81517b0 0x7fc1e8142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fc1e80c7a00 0x7fc1e81f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fc1c806c530 0x7fc1c806e9f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 --2- 172.21.15.132:0/1167826750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e8073610 0x7fc1e8148f20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:14.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 >> 172.21.15.132:0/1167826750 conn(0x7fc1e806fc20 msgr2=0x7fc1e80c9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:14.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.925+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 shutdown_connections 2024-03-27T06:22:14.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:14.926+0000 7fc1edc00640 1 -- 172.21.15.132:0/1167826750 wait complete. 2024-03-27T06:22:14.934 INFO:tasks.ceph:Scrubbing osd.4 2024-03-27T06:22:14.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.206+0000 7f4cdb200640 1 Processor -- start 2024-03-27T06:22:15.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.206+0000 7f4cdb200640 1 -- start start 2024-03-27T06:22:15.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cdb200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 0x7f4cdc0d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cdb200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4cdc0d0620 0x7f4cdc15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cdb200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cdc15df60 con 0x7f4cdc0cc510 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cdb200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f4cdc15ee50 con 0x7f4cdc0d0620 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cdb200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4cdc15e240 con 0x7f4cdc154a30 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 0x7f4cdc0d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cda800640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4cdc0d0620 0x7f4cdc15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4cdc0cc510 0x7f4cdc0cb110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60804/0 (socket says 172.21.15.132:60804) 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9400640 1 -- 172.21.15.132:0/1384688889 learned_addr learned my addr 172.21.15.132:0/1384688889 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:15.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd8a00640 1 -- 172.21.15.132:0/1384688889 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724569042 0 0) 0x7f4cdc15df60 con 0x7f4cdc0cc510 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9e00640 1 -- 172.21.15.132:0/1384688889 >> v2:172.21.15.134:3300/0 conn(0x7f4cdc0d0620 msgr2=0x7f4cdc15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9e00640 1 --2- 172.21.15.132:0/1384688889 >> v2:172.21.15.134:3300/0 conn(0x7f4cdc0d0620 0x7f4cdc15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9e00640 1 -- 172.21.15.132:0/1384688889 >> v1:172.21.15.134:6789/0 conn(0x7f4cdc0cc510 legacy=0x7f4cdc0cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.208+0000 7f4cd9e00640 1 -- 172.21.15.132:0/1384688889 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc15e520 con 0x7f4cdc154a30 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cd9e00640 1 --2- 172.21.15.132:0/1384688889 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 0x7f4cdc0d00e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097a0 tx=0x7f4ccc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74137a34052bae57 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cd8a00640 1 -- 172.21.15.132:0/1384688889 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc035020 con 0x7f4cdc154a30 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cd8a00640 1 -- 172.21.15.132:0/1384688889 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ccc031830 con 0x7f4cdc154a30 2024-03-27T06:22:15.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cd8a00640 1 -- 172.21.15.132:0/1384688889 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc031c00 con 0x7f4cdc154a30 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 -- 172.21.15.132:0/1384688889 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 msgr2=0x7f4cdc0d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 --2- 172.21.15.132:0/1384688889 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 0x7f4cdc0d00e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097a0 tx=0x7f4ccc02b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 -- 172.21.15.132:0/1384688889 shutdown_connections 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 --2- 172.21.15.132:0/1384688889 >> v2:172.21.15.134:3300/0 conn(0x7f4cdc0d0620 0x7f4cdc15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 --2- 172.21.15.132:0/1384688889 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc154a30 0x7f4cdc0d00e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 -- 172.21.15.132:0/1384688889 >> 172.21.15.132:0/1384688889 conn(0x7f4cdc067280 msgr2=0x7f4cdc067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.210+0000 7f4cdb200640 1 -- 172.21.15.132:0/1384688889 shutdown_connections 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.211+0000 7f4cdb200640 1 -- 172.21.15.132:0/1384688889 wait complete. 2024-03-27T06:22:15.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.212+0000 7f4cdb200640 1 Processor -- start 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.212+0000 7f4cdb200640 1 -- start start 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 0x7f4cdc1f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 0x7f4cdc1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cdc154a30 0x7f4cdc0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc154e10 con 0x7f4cdc154a30 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc15eae0 con 0x7f4cdc0d0620 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cdb200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4cdc154f90 con 0x7f4cdc0cc510 2024-03-27T06:22:15.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cd9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 0x7f4cdc1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cd9e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 0x7f4cdc1f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.213+0000 7f4cd9e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 0x7f4cdc1f0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55068/0 (socket says 172.21.15.132:55068) 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9e00640 1 -- 172.21.15.132:0/2311438179 learned_addr learned my addr 172.21.15.132:0/2311438179 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 0x7f4cdc1599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37124/0 (socket says 172.21.15.132:37124) 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cda800640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cdc154a30 0x7f4cdc0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 -- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 msgr2=0x7f4cdc1f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 0x7f4cdc1f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 -- 172.21.15.132:0/2311438179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cdc154a30 msgr2=0x7f4cdc0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cdc154a30 0x7f4cdc0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc150250 con 0x7f4cdc0d0620 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9e00640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4cdc0cc510 0x7f4cdc1f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cda800640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cdc154a30 0x7f4cdc0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd9400640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 0x7f4cdc1599b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800c480 tx=0x7f4cc800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8018070 con 0x7f4cdc0d0620 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.214+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc80074e0 con 0x7f4cdc0d0620 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.215+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cdc1f36a0 con 0x7f4cdc0d0620 2024-03-27T06:22:15.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.215+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8004a20 con 0x7f4cdc0d0620 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.215+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cdc1f3c60 con 0x7f4cdc0d0620 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.215+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4cdc150000 con 0x7f4cdc0d0620 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.217+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4cc801b070 con 0x7f4cdc0d0620 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.217+0000 7f4cd2000640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4cbc06c530 0x7f4cbc06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f4cc808ab00 con 0x7f4cdc0d0620 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd9e00640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4cbc06c530 0x7f4cbc06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd2000640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f4cbc074ab0 0x7f4cbc076ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cc8017ec0 con 0x7f4cbc074ab0 2024-03-27T06:22:15.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4cc808f2a0 con 0x7f4cdc0d0620 2024-03-27T06:22:15.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cda800640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f4cbc074ab0 0x7f4cbc076ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.218+0000 7f4cd9e00640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4cbc06c530 0x7f4cbc06e9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc02b910 tx=0x7f4ccc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.219+0000 7f4cda800640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f4cbc074ab0 0x7f4cbc076ed0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.220+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== osd.4 v2:172.21.15.134:6816/2337158882 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4cc8017ec0 con 0x7f4cbc074ab0 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.234+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 --> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4cdc068730 con 0x7f4cbc074ab0 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.234+0000 7f4cd2000640 1 -- 172.21.15.132:0/2311438179 <== osd.4 v2:172.21.15.134:6816/2337158882 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4cdc068730 con 0x7f4cbc074ab0 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:15.257 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:15.258 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:15.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f4cbc074ab0 msgr2=0x7f4cbc076ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6816/2337158882,v1:172.21.15.134:6817/2337158882] conn(0x7f4cbc074ab0 0x7f4cbc076ed0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4cbc06c530 msgr2=0x7f4cbc06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4cbc06c530 0x7f4cbc06e9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc02b910 tx=0x7f4ccc03c040 comp rx=0 tx=0).stop 2024-03-27T06:22:15.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 msgr2=0x7f4cdc1599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 --2- 172.21.15.132:0/2311438179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cdc0d0620 0x7f4cdc1599b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800c480 tx=0x7f4cc800c940 comp rx=0 tx=0).stop 2024-03-27T06:22:15.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cda800640 1 -- 172.21.15.132:0/2311438179 reap_dead start 2024-03-27T06:22:15.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 shutdown_connections 2024-03-27T06:22:15.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 >> 172.21.15.132:0/2311438179 conn(0x7f4cdc067280 msgr2=0x7f4cdc0b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:15.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 shutdown_connections 2024-03-27T06:22:15.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.235+0000 7f4cdb200640 1 -- 172.21.15.132:0/2311438179 wait complete. 2024-03-27T06:22:15.263 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-27T06:22:15.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.477+0000 7f83e9000640 1 Processor -- start 2024-03-27T06:22:15.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.477+0000 7f83e9000640 1 -- start start 2024-03-27T06:22:15.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.479+0000 7f83e9000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 0x7f83e415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.479+0000 7f83e9000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f83e415b120 0x7f83e40ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.479+0000 7f83e9000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e4158670 con 0x7f83e40c9c80 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.479+0000 7f83e9000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f83e406ed40 con 0x7f83e415b120 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.479+0000 7f83e9000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f83e406e2f0 con 0x7f83e4158290 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e2800640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f83e40c9c80 0x7f83e415e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60816/0 (socket says 172.21.15.132:60816) 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e2800640 1 -- 172.21.15.132:0/1954188166 learned_addr learned my addr 172.21.15.132:0/1954188166 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1e00640 1 --2- 172.21.15.132:0/1954188166 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 0x7f83e415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1400640 1 --2- 172.21.15.132:0/1954188166 >> v2:172.21.15.132:3300/0 conn(0x7f83e415b120 0x7f83e40ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1e00640 1 -- 172.21.15.132:0/1954188166 >> v2:172.21.15.132:3300/0 conn(0x7f83e415b120 msgr2=0x7f83e40ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:15.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1e00640 1 --2- 172.21.15.132:0/1954188166 >> v2:172.21.15.132:3300/0 conn(0x7f83e415b120 0x7f83e40ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1e00640 1 -- 172.21.15.132:0/1954188166 >> v1:172.21.15.134:6789/0 conn(0x7f83e40c9c80 legacy=0x7f83e415e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e1e00640 1 -- 172.21.15.132:0/1954188166 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e406c6d0 con 0x7f83e4158290 2024-03-27T06:22:15.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.480+0000 7f83e0a00640 1 -- 172.21.15.132:0/1954188166 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384880825 0 0) 0x7f83e4158670 con 0x7f83e40c9c80 2024-03-27T06:22:15.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.481+0000 7f83e1e00640 1 --2- 172.21.15.132:0/1954188166 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 0x7f83e415abe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83d40097a0 tx=0x7f83d402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adbe08007709510d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.481+0000 7f83e0a00640 1 -- 172.21.15.132:0/1954188166 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4035020 con 0x7f83e4158290 2024-03-27T06:22:15.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.481+0000 7f83e0a00640 1 -- 172.21.15.132:0/1954188166 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d4031810 con 0x7f83e4158290 2024-03-27T06:22:15.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.481+0000 7f83e0a00640 1 -- 172.21.15.132:0/1954188166 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4031be0 con 0x7f83e4158290 2024-03-27T06:22:15.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 -- 172.21.15.132:0/1954188166 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 msgr2=0x7f83e415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 --2- 172.21.15.132:0/1954188166 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 0x7f83e415abe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83d40097a0 tx=0x7f83d402b8b0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 -- 172.21.15.132:0/1954188166 shutdown_connections 2024-03-27T06:22:15.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 --2- 172.21.15.132:0/1954188166 >> v2:172.21.15.132:3300/0 conn(0x7f83e415b120 0x7f83e40ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 --2- 172.21.15.132:0/1954188166 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e4158290 0x7f83e415abe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 -- 172.21.15.132:0/1954188166 >> 172.21.15.132:0/1954188166 conn(0x7f83e4067280 msgr2=0x7f83e4067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:15.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 -- 172.21.15.132:0/1954188166 shutdown_connections 2024-03-27T06:22:15.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.482+0000 7f83e9000640 1 -- 172.21.15.132:0/1954188166 wait complete. 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.484+0000 7f83e9000640 1 Processor -- start 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.484+0000 7f83e9000640 1 -- start start 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 0x7f83e4147c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.477 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83e406e5d0 con 0x7f83e40b01c0 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f83e406f690 con 0x7f83e40c9c80 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e9000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f83e406f1d0 con 0x7f83e40b2970 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e2800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 0x7f83e4147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37130/0 (socket says 172.21.15.132:37130) 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1e00640 1 -- 172.21.15.132:0/3196513383 learned_addr learned my addr 172.21.15.132:0/3196513383 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:15.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34790/0 (socket says 172.21.15.132:34790) 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.485+0000 7f83e1e00640 1 -- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 msgr2=0x7f83e4147c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1e00640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 0x7f83e4147c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1e00640 1 -- 172.21.15.132:0/3196513383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 msgr2=0x7f83e40b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1e00640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1e00640 1 -- 172.21.15.132:0/3196513383 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e415aa20 con 0x7f83e40c9c80 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1400640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e2800640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 0x7f83e4147c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:15.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e1e00640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f83d40098c0 tx=0x7f83d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4041070 con 0x7f83e40c9c80 2024-03-27T06:22:15.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d403ad00 con 0x7f83e40c9c80 2024-03-27T06:22:15.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e40b2fc0 con 0x7f83e40c9c80 2024-03-27T06:22:15.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4035040 con 0x7f83e40c9c80 2024-03-27T06:22:15.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.486+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e415ac90 con 0x7f83e40c9c80 2024-03-27T06:22:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.487+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e4068840 con 0x7f83e40c9c80 2024-03-27T06:22:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.490+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f83d404c060 con 0x7f83e40c9c80 2024-03-27T06:22:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.490+0000 7f83da000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 0x7f83c406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.490+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f83d40ad760 con 0x7f83e40c9c80 2024-03-27T06:22:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.491+0000 7f83e1400640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 0x7f83c406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.491+0000 7f83e1400640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 0x7f83c406ea20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f83e40c8940 tx=0x7f83d0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.495+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f83d407b4b0 con 0x7f83e40c9c80 2024-03-27T06:22:15.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.753+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f83e406f8f0 con 0x7f83c406c560 2024-03-27T06:22:15.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.754+0000 7f83da000640 1 -- 172.21.15.132:0/3196513383 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f83e406f8f0 con 0x7f83c406c560 2024-03-27T06:22:15.745 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 4 to deep-scrub 2024-03-27T06:22:15.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.757+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 msgr2=0x7f83c406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.757+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 0x7f83c406ea20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f83e40c8940 tx=0x7f83d0009190 comp rx=0 tx=0).stop 2024-03-27T06:22:15.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.757+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 msgr2=0x7f83e4158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.757+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f83d40098c0 tx=0x7f83d40046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 shutdown_connections 2024-03-27T06:22:15.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f83e40b2970 0x7f83e4147c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83e40b01c0 0x7f83e40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f83c406c560 0x7f83c406ea20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 --2- 172.21.15.132:0/3196513383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e40c9c80 0x7f83e4158cf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 >> 172.21.15.132:0/3196513383 conn(0x7f83e4067280 msgr2=0x7f83e40c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:15.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 shutdown_connections 2024-03-27T06:22:15.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.758+0000 7f83e9000640 1 -- 172.21.15.132:0/3196513383 wait complete. 2024-03-27T06:22:15.771 INFO:tasks.ceph:Scrubbing osd.5 2024-03-27T06:22:15.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:15.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.984+0000 7f819f000640 1 Processor -- start 2024-03-27T06:22:15.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.984+0000 7f819f000640 1 -- start start 2024-03-27T06:22:15.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819f000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8198152900 0x7f8198159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819f000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 0x7f81980c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819f000640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8198159740 con 0x7f8198072a50 2024-03-27T06:22:15.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819f000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f81980c83b0 con 0x7f819806c700 2024-03-27T06:22:15.977 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819f000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f81980719b0 con 0x7f8198152900 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f8197e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8198152900 0x7f8198159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f8197400640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8198072a50 0x7f81980c7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:45680/0 (socket says 172.21.15.132:45680) 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f8197400640 1 -- 172.21.15.132:0/2906046669 learned_addr learned my addr 172.21.15.132:0/2906046669 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.986+0000 7f819d000640 1 --2- 172.21.15.132:0/2906046669 >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 0x7f81980c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f8197e00640 1 -- 172.21.15.132:0/2906046669 >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 msgr2=0x7f81980c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f8197e00640 1 --2- 172.21.15.132:0/2906046669 >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 0x7f81980c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f8197e00640 1 -- 172.21.15.132:0/2906046669 >> v1:172.21.15.132:6789/0 conn(0x7f8198072a50 legacy=0x7f81980c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f8197e00640 1 -- 172.21.15.132:0/2906046669 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8198071c90 con 0x7f8198152900 2024-03-27T06:22:15.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f8196a00640 1 -- 172.21.15.132:0/2906046669 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1448579901 0 0) 0x7f8184001110 con 0x7f8198072a50 2024-03-27T06:22:15.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.987+0000 7f819d000640 1 --2- 172.21.15.132:0/2906046669 >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 0x7f81980c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:22:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.988+0000 7f8197e00640 1 --2- 172.21.15.132:0/2906046669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8198152900 0x7f8198159200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8198153040 tx=0x7f818802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0c0fee7ba76cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.988+0000 7f8196a00640 1 -- 172.21.15.132:0/2906046669 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188035020 con 0x7f8198152900 2024-03-27T06:22:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.988+0000 7f8196a00640 1 -- 172.21.15.132:0/2906046669 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188031830 con 0x7f8198152900 2024-03-27T06:22:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.988+0000 7f8196a00640 1 -- 172.21.15.132:0/2906046669 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188031c00 con 0x7f8198152900 2024-03-27T06:22:15.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 -- 172.21.15.132:0/2906046669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8198152900 msgr2=0x7f8198159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 --2- 172.21.15.132:0/2906046669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8198152900 0x7f8198159200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8198153040 tx=0x7f818802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 -- 172.21.15.132:0/2906046669 shutdown_connections 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 --2- 172.21.15.132:0/2906046669 >> v2:172.21.15.132:3300/0 conn(0x7f819806c700 0x7f81980c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 --2- 172.21.15.132:0/2906046669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8198152900 0x7f8198159200 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 -- 172.21.15.132:0/2906046669 >> 172.21.15.132:0/2906046669 conn(0x7f81980c8a80 msgr2=0x7f81980c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 -- 172.21.15.132:0/2906046669 shutdown_connections 2024-03-27T06:22:15.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.989+0000 7f819f000640 1 -- 172.21.15.132:0/2906046669 wait complete. 2024-03-27T06:22:15.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.992+0000 7f819f000640 1 Processor -- start 2024-03-27T06:22:15.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.992+0000 7f819f000640 1 -- start start 2024-03-27T06:22:15.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 0x7f81980c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8198072a50 0x7f81981ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8198152900 0x7f81981eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8198072530 con 0x7f8198072a50 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8198072250 con 0x7f819806c700 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819f000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f819806cb10 con 0x7f8198152900 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f8197e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 0x7f81980c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f8197e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 0x7f81980c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37152/0 (socket says 172.21.15.132:37152) 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f8197e00640 1 -- 172.21.15.132:0/2287228084 learned_addr learned my addr 172.21.15.132:0/2287228084 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:15.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f819d000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8198152900 0x7f81981eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.993+0000 7f8197400640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8198072a50 0x7f81981ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 -- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8198152900 msgr2=0x7f81981eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8198152900 0x7f81981eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 -- 172.21.15.132:0/2287228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8198072a50 msgr2=0x7f81981ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8198072a50 0x7f81981ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81980c6250 con 0x7f819806c700 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197400640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8198072a50 0x7f81981ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T06:22:15.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f8197e00640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 0x7f81980c4540 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8188000c00 tx=0x7f81880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188040070 con 0x7f819806c700 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188039a20 con 0x7f819806c700 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188035040 con 0x7f819806c700 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.994+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81980c5c90 con 0x7f819806c700 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.995+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81980c3a60 con 0x7f819806c700 2024-03-27T06:22:15.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.996+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81981560f0 con 0x7f819806c700 2024-03-27T06:22:15.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.997+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f818804b060 con 0x7f819806c700 2024-03-27T06:22:15.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f818fe00640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f817806c560 0x7f817806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f81880ad0c0 con 0x7f819806c700 2024-03-27T06:22:15.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f8197400640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f817806c560 0x7f817806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f818fe00640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7f8178074af0 0x7f8178076f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81880067f0 con 0x7f8178074af0 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.998+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f81880b4050 con 0x7f819806c700 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.999+0000 7f819d000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7f8178074af0 0x7f8178076f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.999+0000 7f8197400640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f817806c560 0x7f817806ea20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f819814ce50 tx=0x7f8184007a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:15.999+0000 7f819d000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7f8178074af0 0x7f8178076f10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:15.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.000+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== osd.5 v2:172.21.15.134:6808/172932420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81880067f0 con 0x7f8178074af0 2024-03-27T06:22:16.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.015+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 --> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8198068840 con 0x7f8178074af0 2024-03-27T06:22:16.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.015+0000 7f818fe00640 1 -- 172.21.15.132:0/2287228084 <== osd.5 v2:172.21.15.134:6808/172932420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8198068840 con 0x7f8178074af0 2024-03-27T06:22:16.006 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:16.006 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:16.006 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.015+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7f8178074af0 msgr2=0x7f8178076f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6808/172932420,v1:172.21.15.134:6809/172932420] conn(0x7f8178074af0 0x7f8178076f10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f817806c560 msgr2=0x7f817806ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f817806c560 0x7f817806ea20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f819814ce50 tx=0x7f8184007a10 comp rx=0 tx=0).stop 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 msgr2=0x7f81980c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 --2- 172.21.15.132:0/2287228084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819806c700 0x7f81980c4540 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8188000c00 tx=0x7f81880046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819d000640 1 -- 172.21.15.132:0/2287228084 reap_dead start 2024-03-27T06:22:16.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 shutdown_connections 2024-03-27T06:22:16.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 >> 172.21.15.132:0/2287228084 conn(0x7f81980c8a80 msgr2=0x7f8198151b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 shutdown_connections 2024-03-27T06:22:16.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.016+0000 7f819f000640 1 -- 172.21.15.132:0/2287228084 wait complete. 2024-03-27T06:22:16.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-27T06:22:16.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.224+0000 7f26cca00640 1 Processor -- start 2024-03-27T06:22:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.224+0000 7f26cca00640 1 -- start start 2024-03-27T06:22:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.225+0000 7f26cca00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f26c80c7990 0x7f26c8071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.225+0000 7f26cca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.225+0000 7f26cca00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c80c84a0 con 0x7f26c8154d10 2024-03-27T06:22:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.225+0000 7f26cca00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f26c815b610 con 0x7f26c80c7990 2024-03-27T06:22:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.225+0000 7f26cca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f26c815aa40 con 0x7f26c80737e0 2024-03-27T06:22:16.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c6a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f26c80c7990 0x7f26c8071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55118/0 (socket says 172.21.15.132:55118) 2024-03-27T06:22:16.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f26c8154d10 0x7f26c80c9670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60820/0 (socket says 172.21.15.132:60820) 2024-03-27T06:22:16.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 -- 172.21.15.132:0/2911208942 learned_addr learned my addr 172.21.15.132:0/2911208942 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 -- 172.21.15.132:0/2911208942 >> v2:172.21.15.132:3300/0 conn(0x7f26c80c7990 msgr2=0x7f26c8071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:16.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 --2- 172.21.15.132:0/2911208942 >> v2:172.21.15.132:3300/0 conn(0x7f26c80c7990 0x7f26c8071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 -- 172.21.15.132:0/2911208942 >> v1:172.21.15.134:6789/0 conn(0x7f26c8154d10 legacy=0x7f26c80c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c7e00640 1 -- 172.21.15.132:0/2911208942 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c815ace0 con 0x7f26c80737e0 2024-03-27T06:22:16.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.226+0000 7f26c6000640 1 -- 172.21.15.132:0/2911208942 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1969676449 0 0) 0x7f26c80c84a0 con 0x7f26c8154d10 2024-03-27T06:22:16.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.228+0000 7f26c7e00640 1 --2- 172.21.15.132:0/2911208942 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009870 tx=0x7f26b802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb35c0ab9f258989 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.228+0000 7f26c6000640 1 -- 172.21.15.132:0/2911208942 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8035020 con 0x7f26c80737e0 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.228+0000 7f26c6000640 1 -- 172.21.15.132:0/2911208942 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b8031640 con 0x7f26c80737e0 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.228+0000 7f26c6000640 1 -- 172.21.15.132:0/2911208942 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b802d040 con 0x7f26c80737e0 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 -- 172.21.15.132:0/2911208942 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 msgr2=0x7f26c80724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 --2- 172.21.15.132:0/2911208942 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009870 tx=0x7f26b802b8f0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 -- 172.21.15.132:0/2911208942 shutdown_connections 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 --2- 172.21.15.132:0/2911208942 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c80724c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 --2- 172.21.15.132:0/2911208942 >> v2:172.21.15.132:3300/0 conn(0x7f26c80c7990 0x7f26c8071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 -- 172.21.15.132:0/2911208942 >> 172.21.15.132:0/2911208942 conn(0x7f26c806fae0 msgr2=0x7f26c806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 -- 172.21.15.132:0/2911208942 shutdown_connections 2024-03-27T06:22:16.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.229+0000 7f26cca00640 1 -- 172.21.15.132:0/2911208942 wait complete. 2024-03-27T06:22:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.232+0000 7f26cca00640 1 Processor -- start 2024-03-27T06:22:16.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.232+0000 7f26cca00640 1 -- start start 2024-03-27T06:22:16.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c80c7990 0x7f26c815ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26c8154d10 0x7f26c80cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f26c80c9250 con 0x7f26c8154d10 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f26c815b2a0 con 0x7f26c80c7990 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26cca00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f26c80c93d0 con 0x7f26c80737e0 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26c7400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26c7400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55128/0 (socket says 172.21.15.132:55128) 2024-03-27T06:22:16.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26c7400640 1 -- 172.21.15.132:0/2190891324 learned_addr learned my addr 172.21.15.132:0/2190891324 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26c6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c80c7990 0x7f26c815ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.233+0000 7f26c7e00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26c8154d10 0x7f26c80cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 -- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c80c7990 msgr2=0x7f26c815ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c80c7990 0x7f26c815ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 -- 172.21.15.132:0/2190891324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26c8154d10 msgr2=0x7f26c80cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26c8154d10 0x7f26c80cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 -- 172.21.15.132:0/2190891324 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c815e7a0 con 0x7f26c80737e0 2024-03-27T06:22:16.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26c7400640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26c000c500 tx=0x7f26c000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0018070 con 0x7f26c80737e0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26c0007580 con 0x7f26c80737e0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c815ea80 con 0x7f26c80737e0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0004a20 con 0x7f26c80737e0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.234+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c81f07b0 con 0x7f26c80737e0 2024-03-27T06:22:16.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.236+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c80758a0 con 0x7f26c80737e0 2024-03-27T06:22:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.237+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f26c001b070 con 0x7f26c80737e0 2024-03-27T06:22:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.238+0000 7f26bf400640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 0x7f26a406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.238+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f26c008ab00 con 0x7f26c80737e0 2024-03-27T06:22:16.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.238+0000 7f26c6a00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 0x7f26a406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.238+0000 7f26c6a00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 0x7f26a406e9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26c80cdc80 tx=0x7f26b000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.243+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f26c0058970 con 0x7f26c80737e0 2024-03-27T06:22:16.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.511+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f26c80c8830 con 0x7f26a406c530 2024-03-27T06:22:16.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.512+0000 7f26bf400640 1 -- 172.21.15.132:0/2190891324 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f26c80c8830 con 0x7f26a406c530 2024-03-27T06:22:16.502 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 5 to deep-scrub 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 msgr2=0x7f26a406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 0x7f26a406e9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26c80cdc80 tx=0x7f26b000b040 comp rx=0 tx=0).stop 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 msgr2=0x7f26c814f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26c000c500 tx=0x7f26c000c9c0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 shutdown_connections 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26c8154d10 0x7f26c80cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f26a406c530 0x7f26a406e9f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c80c7990 0x7f26c815ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 --2- 172.21.15.132:0/2190891324 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f26c80737e0 0x7f26c814f2d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 >> 172.21.15.132:0/2190891324 conn(0x7f26c806fae0 msgr2=0x7f26c80c47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 shutdown_connections 2024-03-27T06:22:16.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.514+0000 7f26cca00640 1 -- 172.21.15.132:0/2190891324 wait complete. 2024-03-27T06:22:16.521 INFO:tasks.ceph:Scrubbing osd.6 2024-03-27T06:22:16.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:16.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.742+0000 7f5bc2200640 1 Processor -- start 2024-03-27T06:22:16.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.742+0000 7f5bc2200640 1 -- start start 2024-03-27T06:22:16.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc2200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 0x7f5bbc15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc2200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5bbc15df40 0x7f5bbc0cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc2200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bbc15e480 con 0x7f5bbc154e30 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc2200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f5bbc0cba40 con 0x7f5bbc15df40 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc2200640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5bbc15e720 con 0x7f5bbc154a50 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc0e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 0x7f5bbc15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc1800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f5bbc15df40 0x7f5bbc0cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc0e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 0x7f5bbc15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37174/0 (socket says 172.21.15.132:37174) 2024-03-27T06:22:16.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.743+0000 7f5bc0e00640 1 -- 172.21.15.132:0/1020409056 learned_addr learned my addr 172.21.15.132:0/1020409056 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.744+0000 7f5bc1800640 1 -- 172.21.15.132:0/1020409056 >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 msgr2=0x7f5bbc15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:16.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.744+0000 7f5bc1800640 1 --2- 172.21.15.132:0/1020409056 >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 0x7f5bbc15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.744+0000 7f5bc1800640 1 -- 172.21.15.132:0/1020409056 >> v1:172.21.15.134:6789/0 conn(0x7f5bbc154e30 legacy=0x7f5bbc06b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.744+0000 7f5bc1800640 1 -- 172.21.15.132:0/1020409056 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bbc15ea00 con 0x7f5bbc15df40 2024-03-27T06:22:16.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.745+0000 7f5bc1800640 1 --2- 172.21.15.132:0/1020409056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc0cc200 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0cce90 tx=0x7f5bb402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1becb2b731dfedd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.745+0000 7f5bbb400640 1 -- 172.21.15.132:0/1020409056 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4039070 con 0x7f5bbc15df40 2024-03-27T06:22:16.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.745+0000 7f5bbb400640 1 -- 172.21.15.132:0/1020409056 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb402cc70 con 0x7f5bbc15df40 2024-03-27T06:22:16.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.745+0000 7f5bbb400640 1 -- 172.21.15.132:0/1020409056 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4034660 con 0x7f5bbc15df40 2024-03-27T06:22:16.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 -- 172.21.15.132:0/1020409056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 msgr2=0x7f5bbc0cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 --2- 172.21.15.132:0/1020409056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc0cc200 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0cce90 tx=0x7f5bb402bec0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 -- 172.21.15.132:0/1020409056 shutdown_connections 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 --2- 172.21.15.132:0/1020409056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc0cc200 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 --2- 172.21.15.132:0/1020409056 >> v2:172.21.15.134:3300/0 conn(0x7f5bbc154a50 0x7f5bbc15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 -- 172.21.15.132:0/1020409056 >> 172.21.15.132:0/1020409056 conn(0x7f5bbc06d910 msgr2=0x7f5bbc06dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 -- 172.21.15.132:0/1020409056 shutdown_connections 2024-03-27T06:22:16.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.746+0000 7f5bc2200640 1 -- 172.21.15.132:0/1020409056 wait complete. 2024-03-27T06:22:16.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.748+0000 7f5bc2200640 1 Processor -- start 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.748+0000 7f5bc2200640 1 -- start start 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 0x7f5bbc1f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bbc154e30 0x7f5bbc15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc06bfe0 con 0x7f5bbc15df40 2024-03-27T06:22:16.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc0cbce0 con 0x7f5bbc154a50 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc2200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5bbc06c160 con 0x7f5bbc154e30 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 0x7f5bbc1f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 0x7f5bbc1f1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37186/0 (socket says 172.21.15.132:37186) 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bbbe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bbc154e30 0x7f5bbc15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc0e00640 1 -- 172.21.15.132:0/4190030930 learned_addr learned my addr 172.21.15.132:0/4190030930 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.749+0000 7f5bc1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 -- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bbc154e30 msgr2=0x7f5bbc15cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5bbc154e30 0x7f5bbc15cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 -- 172.21.15.132:0/4190030930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 msgr2=0x7f5bbc144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bbc15df40 0x7f5bbc144b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bbc1f1880 con 0x7f5bbc154a50 2024-03-27T06:22:16.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc0e00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 0x7f5bbc1f1340 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000e840 tx=0x7f5bb000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0017070 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb000c8e0 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb000ccd0 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.750+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bbc150560 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.751+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bbc150ae0 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.751+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5bbc068730 con 0x7f5bbc154a50 2024-03-27T06:22:16.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.752+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5bb001a070 con 0x7f5bbc154a50 2024-03-27T06:22:16.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.754+0000 7f5bb8c00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5b9c06c560 0x7f5b9c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.754+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f5bb008b9e0 con 0x7f5bbc154a50 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.754+0000 7f5bbbe00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5b9c06c560 0x7f5b9c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.754+0000 7f5bb8c00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7f5b9c074ae0 0x7f5b9c076f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.755+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bb0016ec0 con 0x7f5b9c074ae0 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.755+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5bb00902a0 con 0x7f5bbc154a50 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.755+0000 7f5bc1800640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7f5b9c074ae0 0x7f5b9c076f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.755+0000 7f5bbbe00640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5b9c06c560 0x7f5b9c06ea20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0cf720 tx=0x7f5ba8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.755+0000 7f5bc1800640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7f5b9c074ae0 0x7f5b9c076f00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.756+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== osd.6 v2:172.21.15.134:6800/3892608742 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5bb0016ec0 con 0x7f5b9c074ae0 2024-03-27T06:22:16.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.771+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 --> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5bbc0ce090 con 0x7f5b9c074ae0 2024-03-27T06:22:16.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.771+0000 7f5bb8c00640 1 -- 172.21.15.132:0/4190030930 <== osd.6 v2:172.21.15.134:6800/3892608742 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5bbc0ce090 con 0x7f5b9c074ae0 2024-03-27T06:22:16.762 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:16.762 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:16.762 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:16.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7f5b9c074ae0 msgr2=0x7f5b9c076f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6800/3892608742,v1:172.21.15.134:6801/3892608742] conn(0x7f5b9c074ae0 0x7f5b9c076f00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5b9c06c560 msgr2=0x7f5b9c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5b9c06c560 0x7f5b9c06ea20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0cf720 tx=0x7f5ba8009040 comp rx=0 tx=0).stop 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 msgr2=0x7f5bbc1f1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 --2- 172.21.15.132:0/4190030930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc154a50 0x7f5bbc1f1340 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000e840 tx=0x7f5bb000ed00 comp rx=0 tx=0).stop 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc1800640 1 -- 172.21.15.132:0/4190030930 reap_dead start 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 shutdown_connections 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 >> 172.21.15.132:0/4190030930 conn(0x7f5bbc06d910 msgr2=0x7f5bbc0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 shutdown_connections 2024-03-27T06:22:16.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.772+0000 7f5bc2200640 1 -- 172.21.15.132:0/4190030930 wait complete. 2024-03-27T06:22:16.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-27T06:22:16.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.985+0000 7f9045000640 1 Processor -- start 2024-03-27T06:22:16.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.985+0000 7f9045000640 1 -- start start 2024-03-27T06:22:16.978 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 0x7f90400c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f90400c77f0 0x7f9040154890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f90400cc210 con 0x7f90400c77f0 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f90400cdad0 con 0x7f904006cc80 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9045000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f904015b690 con 0x7f904006c8a0 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9037e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f903e800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f90400c77f0 0x7f9040154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.979 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f903de00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 0x7f90400c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f903e800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f90400c77f0 0x7f9040154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34852/0 (socket says 172.21.15.132:34852) 2024-03-27T06:22:16.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f9037e00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37202/0 (socket says 172.21.15.132:37202) 2024-03-27T06:22:16.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.988+0000 7f903e800640 1 -- 172.21.15.132:0/3463941333 learned_addr learned my addr 172.21.15.132:0/3463941333 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903e800640 1 -- 172.21.15.132:0/3463941333 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 msgr2=0x7f90400c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:16.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903e800640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 0x7f90400c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903e800640 1 -- 172.21.15.132:0/3463941333 >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 msgr2=0x7f90400c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:16.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903e800640 1 --2- 172.21.15.132:0/3463941333 >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903e800640 1 -- 172.21.15.132:0/3463941333 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904015b970 con 0x7f90400c77f0 2024-03-27T06:22:16.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f9037e00640 1 --2- 172.21.15.132:0/3463941333 >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:16.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.989+0000 7f903de00640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 0x7f90400c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:16.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.990+0000 7f903e800640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f9040154890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9040068260 tx=0x7f903802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d734592bfed17d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f903d400640 1 -- 172.21.15.132:0/3463941333 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903802c9e0 con 0x7f90400c77f0 2024-03-27T06:22:16.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f903d400640 1 -- 172.21.15.132:0/3463941333 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903802cb80 con 0x7f90400c77f0 2024-03-27T06:22:16.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f903d400640 1 -- 172.21.15.132:0/3463941333 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90380343f0 con 0x7f90400c77f0 2024-03-27T06:22:16.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 -- 172.21.15.132:0/3463941333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 msgr2=0x7f9040154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f9040154890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9040068260 tx=0x7f903802be40 comp rx=0 tx=0).stop 2024-03-27T06:22:16.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 -- 172.21.15.132:0/3463941333 shutdown_connections 2024-03-27T06:22:16.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f9040154890 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 --2- 172.21.15.132:0/3463941333 >> v2:172.21.15.134:3300/0 conn(0x7f904006cc80 0x7f90400c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 --2- 172.21.15.132:0/3463941333 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006c8a0 0x7f90400c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 -- 172.21.15.132:0/3463941333 >> 172.21.15.132:0/3463941333 conn(0x7f90400ccbe0 msgr2=0x7f90400ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 -- 172.21.15.132:0/3463941333 shutdown_connections 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.991+0000 7f9045000640 1 -- 172.21.15.132:0/3463941333 wait complete. 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.993+0000 7f9045000640 1 Processor -- start 2024-03-27T06:22:16.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.993+0000 7f9045000640 1 -- start start 2024-03-27T06:22:16.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 0x7f904015bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f90401f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f904006d440 con 0x7f90400c77f0 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f904006d2c0 con 0x7f904006c8a0 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9045000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f90400cd700 con 0x7f904006cc80 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f903de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.994+0000 7f9037e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 0x7f904015bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f903de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37216/0 (socket says 172.21.15.132:37216) 2024-03-27T06:22:16.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f903de00640 1 -- 172.21.15.132:0/923642727 learned_addr learned my addr 172.21.15.132:0/923642727 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f903e800640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f90401f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 -- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 msgr2=0x7f90401f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 -- 172.21.15.132:0/923642727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 msgr2=0x7f90401f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f90401f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 -- 172.21.15.132:0/923642727 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90401f3970 con 0x7f904006cc80 2024-03-27T06:22:16.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f903de00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:16.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9037e00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 0x7f904015bcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f902800e8d0 tx=0x7f902800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.995+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028017070 con 0x7f904006cc80 2024-03-27T06:22:16.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.996+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f902800c940 con 0x7f904006cc80 2024-03-27T06:22:16.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.996+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904015c380 con 0x7f904006cc80 2024-03-27T06:22:16.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.996+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902800cd30 con 0x7f904006cc80 2024-03-27T06:22:16.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.996+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f904015c940 con 0x7f904006cc80 2024-03-27T06:22:16.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.997+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9040068840 con 0x7f904006cc80 2024-03-27T06:22:16.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.998+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f902801a070 con 0x7f904006cc80 2024-03-27T06:22:16.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.999+0000 7f9036000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 0x7f901c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:16.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.999+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f902808bab0 con 0x7f904006cc80 2024-03-27T06:22:16.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:16.999+0000 7f903de00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 0x7f901c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:16.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.000+0000 7f903de00640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 0x7f901c06ea20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9030009560 tx=0x7f9030009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:16.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.005+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9028059780 con 0x7f904006cc80 2024-03-27T06:22:17.258 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.268+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f9040154eb0 con 0x7f901c06c560 2024-03-27T06:22:17.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.269+0000 7f9036000640 1 -- 172.21.15.132:0/923642727 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9040154eb0 con 0x7f901c06c560 2024-03-27T06:22:17.259 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 6 to deep-scrub 2024-03-27T06:22:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.271+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 msgr2=0x7f901c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.271+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 0x7f901c06ea20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9030009560 tx=0x7f9030009190 comp rx=0 tx=0).stop 2024-03-27T06:22:17.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.271+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 msgr2=0x7f904015bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.271+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 0x7f904015bcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f902800e8d0 tx=0x7f902800ed90 comp rx=0 tx=0).stop 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 shutdown_connections 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90400c77f0 0x7f90401f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f904006cc80 0x7f904015bcd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f901c06c560 0x7f901c06ea20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 --2- 172.21.15.132:0/923642727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f904006c8a0 0x7f90401f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 >> 172.21.15.132:0/923642727 conn(0x7f90400ccbe0 msgr2=0x7f90400c8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 shutdown_connections 2024-03-27T06:22:17.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.272+0000 7f9045000640 1 -- 172.21.15.132:0/923642727 wait complete. 2024-03-27T06:22:17.289 INFO:tasks.ceph:Scrubbing osd.7 2024-03-27T06:22:17.289 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-27T06:22:17.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.510+0000 7f8dcc600640 1 Processor -- start 2024-03-27T06:22:17.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.511+0000 7f8dcc600640 1 -- start start 2024-03-27T06:22:17.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dcc600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8dc40cdfa0 0x7f8dc40ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dcc600640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc4159270 con 0x7f8dc406f9c0 2024-03-27T06:22:17.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dcc600640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc406f540 con 0x7f8dc40c82d0 2024-03-27T06:22:17.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dcc600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f8dc40c3740 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dc9200640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8dc40c82d0 0x7f8dc40c3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60830/0 (socket says 172.21.15.132:60830) 2024-03-27T06:22:17.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dc9c00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f8dc40cdfa0 0x7f8dc40ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.512+0000 7f8dc9200640 1 -- 172.21.15.132:0/1650327885 learned_addr learned my addr 172.21.15.132:0/1650327885 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853718968 0 0) 0x7f8dc406f540 con 0x7f8dc40c82d0 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dac003520 con 0x7f8dc40c82d0 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc9c00640 1 -- 172.21.15.132:0/1650327885 >> v1:172.21.15.134:6789/0 conn(0x7f8dc40c82d0 legacy=0x7f8dc40c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804936710 0 0) 0x7f8dc4159270 con 0x7f8dc406f9c0 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc9c00640 1 -- 172.21.15.132:0/1650327885 >> v1:172.21.15.132:6789/0 conn(0x7f8dc406f9c0 legacy=0x7f8dc4162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.513+0000 7f8dc9c00640 1 -- 172.21.15.132:0/1650327885 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc415b9b0 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.514+0000 7f8dc9c00640 1 --2- 172.21.15.132:0/1650327885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc40ce380 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097a0 tx=0x7f8db802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a24c6fe4077d8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.515+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8035020 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.515+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db8031830 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.515+0000 7f8dc3e00640 1 -- 172.21.15.132:0/1650327885 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8031c00 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 -- 172.21.15.132:0/1650327885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40cdfa0 msgr2=0x7f8dc40ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 --2- 172.21.15.132:0/1650327885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc40ce380 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8db80097a0 tx=0x7f8db802b8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 -- 172.21.15.132:0/1650327885 shutdown_connections 2024-03-27T06:22:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 --2- 172.21.15.132:0/1650327885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc40ce380 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 -- 172.21.15.132:0/1650327885 >> 172.21.15.132:0/1650327885 conn(0x7f8dc4067280 msgr2=0x7f8dc4067690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 -- 172.21.15.132:0/1650327885 shutdown_connections 2024-03-27T06:22:17.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.516+0000 7f8dcc600640 1 -- 172.21.15.132:0/1650327885 wait complete. 2024-03-27T06:22:17.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.519+0000 7f8dcc600640 1 Processor -- start 2024-03-27T06:22:17.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.519+0000 7f8dcc600640 1 -- start start 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 0x7f8dc41ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40c82d0 0x7f8dc40cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc4140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc40c4de0 con 0x7f8dc40c82d0 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc406ebf0 con 0x7f8dc40cdfa0 2024-03-27T06:22:17.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dcc600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc40c50c0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 0x7f8dc41ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 0x7f8dc41ec1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55172/0 (socket says 172.21.15.132:55172) 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 -- 172.21.15.132:0/2112341139 learned_addr learned my addr 172.21.15.132:0/2112341139 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9200640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40c82d0 0x7f8dc40cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dca600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc4140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 -- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dc40cdfa0 msgr2=0x7f8dc4140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dc40cdfa0 0x7f8dc4140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 -- 172.21.15.132:0/2112341139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40c82d0 msgr2=0x7f8dc40cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc40c82d0 0x7f8dc40cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.521+0000 7f8dc9c00640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc41ec6e0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dc9c00640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 0x7f8dc41ec1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8db802bee0 tx=0x7f8db80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db803a6f0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db803a8b0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db803aca0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc414a060 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.522+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc41ef800 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.523+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8dc414a600 con 0x7f8dc406f9c0 2024-03-27T06:22:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.525+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f8db8038020 con 0x7f8dc406f9c0 2024-03-27T06:22:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc1600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8da406c530 0x7f8da406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f8db80ad1e0 con 0x7f8dc406f9c0 2024-03-27T06:22:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc9200640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8da406c530 0x7f8da406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc1600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f8da4074ab0 0x7f8da4076ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8db8030e50 con 0x7f8da4074ab0 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.526+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8db80b4050 con 0x7f8dc406f9c0 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.527+0000 7f8dca600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f8da4074ab0 0x7f8da4076ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.527+0000 7f8dc9200640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8da406c530 0x7f8da406e9f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cd440 tx=0x7f8db0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.527+0000 7f8dca600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f8da4074ab0 0x7f8da4076ed0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.528+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== osd.7 v2:172.21.15.134:6824/789010038 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8db8030e50 con 0x7f8da4074ab0 2024-03-27T06:22:17.540 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.550+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 --> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8dc4068840 con 0x7f8da4074ab0 2024-03-27T06:22:17.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.550+0000 7f8dc1600640 1 -- 172.21.15.132:0/2112341139 <== osd.7 v2:172.21.15.134:6824/789010038 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8dc4068840 con 0x7f8da4074ab0 2024-03-27T06:22:17.541 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-27T06:22:17.542 INFO:teuthology.orchestra.run.smithi132.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T06:22:17.542 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f8da4074ab0 msgr2=0x7f8da4076ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6824/789010038,v1:172.21.15.134:6825/789010038] conn(0x7f8da4074ab0 0x7f8da4076ed0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8da406c530 msgr2=0x7f8da406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f8da406c530 0x7f8da406e9f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cd440 tx=0x7f8db0007d30 comp rx=0 tx=0).stop 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 msgr2=0x7f8dc41ec1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 --2- 172.21.15.132:0/2112341139 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f8dc406f9c0 0x7f8dc41ec1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8db802bee0 tx=0x7f8db80046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dca600640 1 -- 172.21.15.132:0/2112341139 reap_dead start 2024-03-27T06:22:17.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 shutdown_connections 2024-03-27T06:22:17.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 >> 172.21.15.132:0/2112341139 conn(0x7f8dc4067280 msgr2=0x7f8dc4157970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:17.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.551+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 shutdown_connections 2024-03-27T06:22:17.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.552+0000 7f8dcc600640 1 -- 172.21.15.132:0/2112341139 wait complete. 2024-03-27T06:22:17.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-27T06:22:17.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.770+0000 7f0d7a600640 1 Processor -- start 2024-03-27T06:22:17.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.771+0000 7f0d7a600640 1 -- start start 2024-03-27T06:22:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0d741549a0 0x7f0d740cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 0x7f0d7406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f0d740cada0 con 0x7f0d740cb870 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7415de10 con 0x7f0d741549a0 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d7a600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7415e0b0 con 0x7f0d7406be10 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d79200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0d741549a0 0x7f0d740cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d73e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 0x7f0d7406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.772+0000 7f0d79c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0d741549a0 0x7f0d740cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:37232/0 (socket says 172.21.15.132:37232) 2024-03-27T06:22:17.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d73e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 0x7f0d7406b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34878/0 (socket says 172.21.15.132:34878) 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 -- 172.21.15.132:0/3157291615 learned_addr learned my addr 172.21.15.132:0/3157291615 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79c00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55176/0 (socket says 172.21.15.132:55176) 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 -- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 msgr2=0x7f0d7415adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 -- 172.21.15.132:0/3157291615 >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 msgr2=0x7f0d7406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 --2- 172.21.15.132:0/3157291615 >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 0x7f0d7406b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79200640 1 -- 172.21.15.132:0/3157291615 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7415e390 con 0x7f0d741549a0 2024-03-27T06:22:17.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.773+0000 7f0d79c00640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:17.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.774+0000 7f0d79200640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d741549a0 0x7f0d740cd6a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0d740c73e0 tx=0x7f0d6802b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e894e77d6d83e37c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.774+0000 7f0d73400640 1 -- 172.21.15.132:0/3157291615 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68034070 con 0x7f0d741549a0 2024-03-27T06:22:17.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.774+0000 7f0d73400640 1 -- 172.21.15.132:0/3157291615 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d68030850 con 0x7f0d741549a0 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.774+0000 7f0d73400640 1 -- 172.21.15.132:0/3157291615 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68030c20 con 0x7f0d741549a0 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 -- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d741549a0 msgr2=0x7f0d740cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d741549a0 0x7f0d740cd6a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0d740c73e0 tx=0x7f0d6802b990 comp rx=0 tx=0).stop 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 -- 172.21.15.132:0/3157291615 shutdown_connections 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7415adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 --2- 172.21.15.132:0/3157291615 >> v2:172.21.15.132:3300/0 conn(0x7f0d740cb870 0x7f0d7406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 --2- 172.21.15.132:0/3157291615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d741549a0 0x7f0d740cd6a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 -- 172.21.15.132:0/3157291615 >> 172.21.15.132:0/3157291615 conn(0x7f0d7406d8d0 msgr2=0x7f0d7406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:17.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 -- 172.21.15.132:0/3157291615 shutdown_connections 2024-03-27T06:22:17.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.775+0000 7f0d7a600640 1 -- 172.21.15.132:0/3157291615 wait complete. 2024-03-27T06:22:17.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.777+0000 7f0d7a600640 1 Processor -- start 2024-03-27T06:22:17.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.777+0000 7f0d7a600640 1 -- start start 2024-03-27T06:22:17.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 0x7f0d740d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d741549a0 0x7f0d7414cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7415ec30 con 0x7f0d741549a0 2024-03-27T06:22:17.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7415e950 con 0x7f0d740cb870 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d7a600640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7415a820 con 0x7f0d7406be10 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d79200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 0x7f0d740d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d79200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55190/0 (socket says 172.21.15.132:55190) 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d79200640 1 -- 172.21.15.132:0/4136196637 learned_addr learned my addr 172.21.15.132:0/4136196637 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.778+0000 7f0d79c00640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d741549a0 0x7f0d7414cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 -- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 msgr2=0x7f0d740d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 0x7f0d740d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 -- 172.21.15.132:0/4136196637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d741549a0 msgr2=0x7f0d7414cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d741549a0 0x7f0d7414cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 -- 172.21.15.132:0/4136196637 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d741589f0 con 0x7f0d7406be10 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d73e00640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 0x7f0d740d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d79200640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0d6802b9d0 tx=0x7f0d68004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68048070 con 0x7f0d7406be10 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d68039a10 con 0x7f0d7406be10 2024-03-27T06:22:17.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68034040 con 0x7f0d7406be10 2024-03-27T06:22:17.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.779+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d74158c90 con 0x7f0d7406be10 2024-03-27T06:22:17.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.780+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d740b5280 con 0x7f0d7406be10 2024-03-27T06:22:17.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.781+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d740686c0 con 0x7f0d7406be10 2024-03-27T06:22:17.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.782+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f0d6804b070 con 0x7f0d7406be10 2024-03-27T06:22:17.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.782+0000 7f0d70c00640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 0x7f0d5406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:17.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.783+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f0d680ad770 con 0x7f0d7406be10 2024-03-27T06:22:17.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.783+0000 7f0d73e00640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 0x7f0d5406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:17.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.783+0000 7f0d73e00640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 0x7f0d5406e9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0d741f1090 tx=0x7f0d60009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:17.782 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.791+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f0d6807b4e0 con 0x7f0d7406be10 2024-03-27T06:22:17.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.997+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f0d740cafd0 con 0x7f0d5406c530 2024-03-27T06:22:17.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:17.998+0000 7f0d70c00640 1 -- 172.21.15.132:0/4136196637 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0d740cafd0 con 0x7f0d5406c530 2024-03-27T06:22:17.988 INFO:teuthology.orchestra.run.smithi132.stderr:instructed osd(s) 7 to deep-scrub 2024-03-27T06:22:17.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.000+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 msgr2=0x7f0d5406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.000+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 0x7f0d5406e9f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0d741f1090 tx=0x7f0d60009190 comp rx=0 tx=0).stop 2024-03-27T06:22:17.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.000+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 msgr2=0x7f0d7414b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:17.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.000+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0d6802b9d0 tx=0x7f0d68004730 comp rx=0 tx=0).stop 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 shutdown_connections 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d741549a0 0x7f0d7414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f0d5406c530 0x7f0d5406e9f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d740cb870 0x7f0d740d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 --2- 172.21.15.132:0/4136196637 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f0d7406be10 0x7f0d7414b980 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 >> 172.21.15.132:0/4136196637 conn(0x7f0d7406d8d0 msgr2=0x7f0d74152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:17.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 shutdown_connections 2024-03-27T06:22:17.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.001+0000 7f0d7a600640 1 -- 172.21.15.132:0/4136196637 wait complete. 2024-03-27T06:22:18.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:22:18.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.224+0000 7f4d35a00640 1 Processor -- start 2024-03-27T06:22:18.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.224+0000 7f4d35a00640 1 -- start start 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4d301549a0 0x7f4d300cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d3015adb0 0x7f4d3015b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3015de10 con 0x7f4d300cb870 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f4d300cada0 con 0x7f4d301549a0 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3015e0b0 con 0x7f4d3015adb0 2024-03-27T06:22:18.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d2fe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4d301549a0 0x7f4d300cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d2f400640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4d300cb870 0x7f4d3006b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:60844/0 (socket says 172.21.15.132:60844) 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d2f400640 1 -- 172.21.15.132:0/3204208765 learned_addr learned my addr 172.21.15.132:0/3204208765 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.226+0000 7f4d35000640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d3015adb0 0x7f4d3015b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.227+0000 7f4d2fe00640 1 -- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d3015adb0 msgr2=0x7f4d3015b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.227+0000 7f4d2fe00640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d3015adb0 0x7f4d3015b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.227+0000 7f4d2fe00640 1 -- 172.21.15.132:0/3204208765 >> v1:172.21.15.134:6789/0 conn(0x7f4d300cb870 legacy=0x7f4d3006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.227+0000 7f4d2fe00640 1 -- 172.21.15.132:0/3204208765 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3015e390 con 0x7f4d301549a0 2024-03-27T06:22:18.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.227+0000 7f4d2ea00640 1 -- 172.21.15.132:0/3204208765 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194905652 0 0) 0x7f4d3015de10 con 0x7f4d300cb870 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.228+0000 7f4d2fe00640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d301549a0 0x7f4d300cd6a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4d200097a0 tx=0x7f4d2002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=295be60cfb49d47b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.228+0000 7f4d2ea00640 1 -- 172.21.15.132:0/3204208765 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20034070 con 0x7f4d301549a0 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.228+0000 7f4d2ea00640 1 -- 172.21.15.132:0/3204208765 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d20030610 con 0x7f4d301549a0 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.228+0000 7f4d2ea00640 1 -- 172.21.15.132:0/3204208765 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200309e0 con 0x7f4d301549a0 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 -- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d301549a0 msgr2=0x7f4d300cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d301549a0 0x7f4d300cd6a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4d200097a0 tx=0x7f4d2002b960 comp rx=0 tx=0).stop 2024-03-27T06:22:18.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 -- 172.21.15.132:0/3204208765 shutdown_connections 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d3015adb0 0x7f4d3015b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 --2- 172.21.15.132:0/3204208765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d301549a0 0x7f4d300cd6a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 -- 172.21.15.132:0/3204208765 >> 172.21.15.132:0/3204208765 conn(0x7f4d3006d8d0 msgr2=0x7f4d3006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 -- 172.21.15.132:0/3204208765 shutdown_connections 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.229+0000 7f4d35a00640 1 -- 172.21.15.132:0/3204208765 wait complete. 2024-03-27T06:22:18.221 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.232+0000 7f4d35a00640 1 Processor -- start 2024-03-27T06:22:18.222 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.232+0000 7f4d35a00640 1 -- start start 2024-03-27T06:22:18.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 0x7f4d301f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 0x7f4d301f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3015ec30 con 0x7f4d301549a0 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3015e950 con 0x7f4d3015adb0 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d35a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3015b700 con 0x7f4d300cb870 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.233+0000 7f4d2fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:55206/0 (socket says 172.21.15.132:55206) 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 -- 172.21.15.132:0/1781122885 learned_addr learned my addr 172.21.15.132:0/1781122885 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 0x7f4d301f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d35000640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 0x7f4d301f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.225 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 -- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 msgr2=0x7f4d301f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 0x7f4d301f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 -- 172.21.15.132:0/1781122885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 msgr2=0x7f4d301f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 0x7f4d301f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 -- 172.21.15.132:0/1781122885 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3014b9f0 con 0x7f4d300cb870 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d35000640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 0x7f4d301f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:18.226 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2f400640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 0x7f4d301f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:18.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.234+0000 7f4d2fe00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d200098c0 tx=0x7f4d200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:18.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.235+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20034040 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.235+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d20030410 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.235+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3014ef80 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.235+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20039c70 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.235+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3014bf70 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.236+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d300686c0 con 0x7f4d300cb870 2024-03-27T06:22:18.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.237+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4d20037070 con 0x7f4d300cb870 2024-03-27T06:22:18.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.238+0000 7f4d27e00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 0x7f4d1006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:18.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.239+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f4d200ad750 con 0x7f4d300cb870 2024-03-27T06:22:18.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.239+0000 7f4d2f400640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 0x7f4d1006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:18.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.239+0000 7f4d2f400640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 0x7f4d1006e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4d3006e0b0 tx=0x7f4d1c007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.244+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4d2007b540 con 0x7f4d300cb870 2024-03-27T06:22:18.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.495+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d300cafd0 con 0x7f4d1006c530 2024-03-27T06:22:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.503+0000 7f4d27e00640 1 -- 172.21.15.132:0/1781122885 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255085 (secure 0 0 0) 0x7f4d300cafd0 con 0x7f4d1006c530 2024-03-27T06:22:18.496 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:18.498 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:22:18.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 msgr2=0x7f4d1006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 0x7f4d1006e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4d3006e0b0 tx=0x7f4d1c007de0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 msgr2=0x7f4d3014d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d200098c0 tx=0x7f4d200046c0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 shutdown_connections 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4d1006c530 0x7f4d1006e9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d3015adb0 0x7f4d301f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d301549a0 0x7f4d301f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 --2- 172.21.15.132:0/1781122885 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4d300cb870 0x7f4d3014d2a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 >> 172.21.15.132:0/1781122885 conn(0x7f4d3006d8d0 msgr2=0x7f4d300cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:18.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 shutdown_connections 2024-03-27T06:22:18.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:18.508+0000 7f4d35a00640 1 -- 172.21.15.132:0/1781122885 wait complete. 2024-03-27T06:22:18.517 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-03-27T06:22:18.482761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1124190479,"num_objects":10759,"num_object_clones":0,"num_object_copies":21518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10759,"num_whiteouts":0,"num_read":1825,"num_read_kb":514907,"num_write":25443,"num_write_kb":1251090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":97,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":62720,"num_omap_keys":108,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26024,"ondisk_log_size":26024,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2133112,"kb_used_data":1755960,"kb_used_omap":12,"kb_used_meta":377075,"kb_avail":752841608,"statfs":{"total":773094113280,"available":770909806592,"internally_reserved":0,"allocated":1798103040,"data_stored":2248610702,"data_compressed":214755512,"data_compressed_allocated":244908032,"data_compressed_original":753008640,"omap_allocated":12751,"internal_metadata":386125361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":484993,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":62720,"num_omap_keys":108,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001933"},"pg_stats":[{"pgid":"3.3f","version":"44'339","reported_seq":2443,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.724989+0000","last_change":"2024-03-27T06:18:40.841294+0000","last_active":"2024-03-27T06:22:08.724989+0000","last_peered":"2024-03-27T06:22:08.724989+0000","last_clean":"2024-03-27T06:22:08.724989+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:22:08.724989+0000","last_undegraded":"2024-03-27T06:22:08.724989+0000","last_fullsized":"2024-03-27T06:22:08.724989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048207270000000003,"stat_sum":{"num_bytes":26656138,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":18,"num_read_kb":8207,"num_write":331,"num_write_kb":26201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"44'318","reported_seq":2419,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671632+0000","last_change":"2024-03-27T06:18:40.829515+0000","last_active":"2024-03-27T06:22:08.671632+0000","last_peered":"2024-03-27T06:22:08.671632+0000","last_clean":"2024-03-27T06:22:08.671632+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:22:08.671632+0000","last_undegraded":"2024-03-27T06:22:08.671632+0000","last_fullsized":"2024-03-27T06:22:08.671632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:20.543774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037350370000000001,"stat_sum":{"num_bytes":5169603,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":315,"num_write_kb":5186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"44'335","reported_seq":2438,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.672297+0000","last_change":"2024-03-27T06:18:40.829898+0000","last_active":"2024-03-27T06:22:08.672297+0000","last_peered":"2024-03-27T06:22:08.672297+0000","last_clean":"2024-03-27T06:22:08.672297+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:22:08.672297+0000","last_undegraded":"2024-03-27T06:22:08.672297+0000","last_fullsized":"2024-03-27T06:22:08.672297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:33:05.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036092289999999998,"stat_sum":{"num_bytes":13816404,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":21,"num_read_kb":8211,"num_write":321,"num_write_kb":21844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"44'316","reported_seq":2416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.679505+0000","last_change":"2024-03-27T06:18:40.562894+0000","last_active":"2024-03-27T06:22:08.679505+0000","last_peered":"2024-03-27T06:22:08.679505+0000","last_clean":"2024-03-27T06:22:08.679505+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:22:08.679505+0000","last_undegraded":"2024-03-27T06:22:08.679505+0000","last_fullsized":"2024-03-27T06:22:08.679505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002773321,"stat_sum":{"num_bytes":19633039,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":13,"num_read_kb":8202,"num_write":307,"num_write_kb":19330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"44'354","reported_seq":2459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.696302+0000","last_change":"2024-03-27T06:18:40.840112+0000","last_active":"2024-03-27T06:22:08.696302+0000","last_peered":"2024-03-27T06:22:08.696302+0000","last_clean":"2024-03-27T06:22:08.696302+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:22:08.696302+0000","last_undegraded":"2024-03-27T06:22:08.696302+0000","last_fullsized":"2024-03-27T06:22:08.696302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071357900000000004,"stat_sum":{"num_bytes":5850441,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":350,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"44'326","reported_seq":2432,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.678660+0000","last_change":"2024-03-27T06:18:40.562623+0000","last_active":"2024-03-27T06:22:08.678660+0000","last_peered":"2024-03-27T06:22:08.678660+0000","last_clean":"2024-03-27T06:22:08.678660+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:22:08.678660+0000","last_undegraded":"2024-03-27T06:22:08.678660+0000","last_fullsized":"2024-03-27T06:22:08.678660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002466584,"stat_sum":{"num_bytes":14040535,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":20,"num_read_kb":3571,"num_write":321,"num_write_kb":13825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"44'356","reported_seq":2460,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.691244+0000","last_change":"2024-03-27T06:18:40.838197+0000","last_active":"2024-03-27T06:22:08.691244+0000","last_peered":"2024-03-27T06:22:08.691244+0000","last_clean":"2024-03-27T06:22:08.691244+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:22:08.691244+0000","last_undegraded":"2024-03-27T06:22:08.691244+0000","last_fullsized":"2024-03-27T06:22:08.691244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056884500000000003,"stat_sum":{"num_bytes":24259091,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":18,"num_read_kb":4113,"num_write":354,"num_write_kb":23807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"44'332","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.691807+0000","last_change":"2024-03-27T06:18:40.830353+0000","last_active":"2024-03-27T06:22:08.691807+0000","last_peered":"2024-03-27T06:22:08.691807+0000","last_clean":"2024-03-27T06:22:08.691807+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:22:08.691807+0000","last_undegraded":"2024-03-27T06:22:08.691807+0000","last_fullsized":"2024-03-27T06:22:08.691807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:33:36.388754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033930689999999999,"stat_sum":{"num_bytes":15911530,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":12,"num_read_kb":4107,"num_write":326,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"44'331","reported_seq":2436,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.725239+0000","last_change":"2024-03-27T06:18:40.834747+0000","last_active":"2024-03-27T06:22:08.725239+0000","last_peered":"2024-03-27T06:22:08.725239+0000","last_clean":"2024-03-27T06:22:08.725239+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:22:08.725239+0000","last_undegraded":"2024-03-27T06:22:08.725239+0000","last_fullsized":"2024-03-27T06:22:08.725239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054729009999999996,"stat_sum":{"num_bytes":6593503,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":20,"num_read_kb":4119,"num_write":318,"num_write_kb":10665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"44'323","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.694224+0000","last_change":"2024-03-27T06:18:40.841044+0000","last_active":"2024-03-27T06:22:08.694224+0000","last_peered":"2024-03-27T06:22:08.694224+0000","last_clean":"2024-03-27T06:22:08.694224+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:22:08.694224+0000","last_undegraded":"2024-03-27T06:22:08.694224+0000","last_fullsized":"2024-03-27T06:22:08.694224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068694350000000001,"stat_sum":{"num_bytes":18722866,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":20,"num_read_kb":9776,"num_write":307,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"44'304","reported_seq":2411,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.713647+0000","last_change":"2024-03-27T06:18:40.840622+0000","last_active":"2024-03-27T06:22:08.713647+0000","last_peered":"2024-03-27T06:22:08.713647+0000","last_clean":"2024-03-27T06:22:08.713647+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:22:08.713647+0000","last_undegraded":"2024-03-27T06:22:08.713647+0000","last_fullsized":"2024-03-27T06:22:08.713647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070900370000000004,"stat_sum":{"num_bytes":18171100,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":21,"num_read_kb":8210,"num_write":297,"num_write_kb":17901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"44'304","reported_seq":2399,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.697939+0000","last_change":"2024-03-27T06:18:40.840141+0000","last_active":"2024-03-27T06:22:08.697939+0000","last_peered":"2024-03-27T06:22:08.697939+0000","last_clean":"2024-03-27T06:22:08.697939+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:22:08.697939+0000","last_undegraded":"2024-03-27T06:22:08.697939+0000","last_fullsized":"2024-03-27T06:22:08.697939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072900689999999997,"stat_sum":{"num_bytes":8281717,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":296,"num_write_kb":12341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"44'301","reported_seq":2406,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.683907+0000","last_change":"2024-03-27T06:18:40.826940+0000","last_active":"2024-03-27T06:22:08.683907+0000","last_peered":"2024-03-27T06:22:08.683907+0000","last_clean":"2024-03-27T06:22:08.683907+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:22:08.683907+0000","last_undegraded":"2024-03-27T06:22:08.683907+0000","last_fullsized":"2024-03-27T06:22:08.683907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":301,"log_dups_size":0,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.552211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029554939999999999,"stat_sum":{"num_bytes":19210972,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":21,"num_read_kb":4116,"num_write":287,"num_write_kb":27094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"44'342","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.653404+0000","last_change":"2024-03-27T06:18:40.827478+0000","last_active":"2024-03-27T06:22:08.653404+0000","last_peered":"2024-03-27T06:22:08.653404+0000","last_clean":"2024-03-27T06:22:08.653404+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:22:08.653404+0000","last_undegraded":"2024-03-27T06:22:08.653404+0000","last_fullsized":"2024-03-27T06:22:08.653404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:34:11.809253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003176613,"stat_sum":{"num_bytes":7395932,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":20,"num_read_kb":4114,"num_write":334,"num_write_kb":11493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"44'356","reported_seq":2461,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.692807+0000","last_change":"2024-03-27T06:18:40.829238+0000","last_active":"2024-03-27T06:22:08.692807+0000","last_peered":"2024-03-27T06:22:08.692807+0000","last_clean":"2024-03-27T06:22:08.692807+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:22:08.692807+0000","last_undegraded":"2024-03-27T06:22:08.692807+0000","last_fullsized":"2024-03-27T06:22:08.692807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:21:38.019283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036973140000000002,"stat_sum":{"num_bytes":23111157,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":349,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"44'358","reported_seq":2464,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.716138+0000","last_change":"2024-03-27T06:18:40.826025+0000","last_active":"2024-03-27T06:22:08.716138+0000","last_peered":"2024-03-27T06:22:08.716138+0000","last_clean":"2024-03-27T06:22:08.716138+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:22:08.716138+0000","last_undegraded":"2024-03-27T06:22:08.716138+0000","last_fullsized":"2024-03-27T06:22:08.716138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001289739,"stat_sum":{"num_bytes":12029261,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":347,"num_write_kb":15208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"44'347","reported_seq":2454,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.724531+0000","last_change":"2024-03-27T06:18:40.835662+0000","last_active":"2024-03-27T06:22:08.724531+0000","last_peered":"2024-03-27T06:22:08.724531+0000","last_clean":"2024-03-27T06:22:08.724531+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:22:08.724531+0000","last_undegraded":"2024-03-27T06:22:08.724531+0000","last_fullsized":"2024-03-27T06:22:08.724531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089932469999999994,"stat_sum":{"num_bytes":10346712,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":335,"num_write_kb":14376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"44'313","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.661964+0000","last_change":"2024-03-27T06:18:40.826258+0000","last_active":"2024-03-27T06:22:08.661964+0000","last_peered":"2024-03-27T06:22:08.661964+0000","last_clean":"2024-03-27T06:22:08.661964+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:22:08.661964+0000","last_undegraded":"2024-03-27T06:22:08.661964+0000","last_fullsized":"2024-03-27T06:22:08.661964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:50:38.551582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010652260000000001,"stat_sum":{"num_bytes":8103937,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":302,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"44'326","reported_seq":2434,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.718176+0000","last_change":"2024-03-27T06:18:40.836497+0000","last_active":"2024-03-27T06:22:08.718176+0000","last_peered":"2024-03-27T06:22:08.718176+0000","last_clean":"2024-03-27T06:22:08.718176+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:22:08.718176+0000","last_undegraded":"2024-03-27T06:22:08.718176+0000","last_fullsized":"2024-03-27T06:22:08.718176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075917839999999999,"stat_sum":{"num_bytes":14133748,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":23,"num_read_kb":12737,"num_write":315,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"44'353","reported_seq":2462,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.697948+0000","last_change":"2024-03-27T06:18:40.840351+0000","last_active":"2024-03-27T06:22:08.697948+0000","last_peered":"2024-03-27T06:22:08.697948+0000","last_clean":"2024-03-27T06:22:08.697948+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:22:08.697948+0000","last_undegraded":"2024-03-27T06:22:08.697948+0000","last_fullsized":"2024-03-27T06:22:08.697948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072459589999999997,"stat_sum":{"num_bytes":23160385,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":24,"num_read_kb":12308,"num_write":342,"num_write_kb":30986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"44'346","reported_seq":2457,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.666298+0000","last_change":"2024-03-27T06:18:40.562416+0000","last_active":"2024-03-27T06:22:08.666298+0000","last_peered":"2024-03-27T06:22:08.666298+0000","last_clean":"2024-03-27T06:22:08.666298+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:22:08.666298+0000","last_undegraded":"2024-03-27T06:22:08.666298+0000","last_fullsized":"2024-03-27T06:22:08.666298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001666102,"stat_sum":{"num_bytes":14662751,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":26,"num_read_kb":4119,"num_write":336,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"44'337","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.696662+0000","last_change":"2024-03-27T06:18:40.829281+0000","last_active":"2024-03-27T06:22:08.696662+0000","last_peered":"2024-03-27T06:22:08.696662+0000","last_clean":"2024-03-27T06:22:08.696662+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:22:08.696662+0000","last_undegraded":"2024-03-27T06:22:08.696662+0000","last_fullsized":"2024-03-27T06:22:08.696662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024688459999999998,"stat_sum":{"num_bytes":9427787,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":13475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"44'337","reported_seq":2450,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.678916+0000","last_change":"2024-03-27T06:18:40.829763+0000","last_active":"2024-03-27T06:22:08.678916+0000","last_peered":"2024-03-27T06:22:08.678916+0000","last_clean":"2024-03-27T06:22:08.678916+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:22:08.678916+0000","last_undegraded":"2024-03-27T06:22:08.678916+0000","last_fullsized":"2024-03-27T06:22:08.678916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:39:51.715239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039259519999999999,"stat_sum":{"num_bytes":16419938,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":29,"num_read_kb":18059,"num_write":326,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"44'348","reported_seq":2452,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.716077+0000","last_change":"2024-03-27T06:18:40.826230+0000","last_active":"2024-03-27T06:22:08.716077+0000","last_peered":"2024-03-27T06:22:08.716077+0000","last_clean":"2024-03-27T06:22:08.716077+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:22:08.716077+0000","last_undegraded":"2024-03-27T06:22:08.716077+0000","last_fullsized":"2024-03-27T06:22:08.716077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470573,"stat_sum":{"num_bytes":19403206,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":18,"num_read_kb":8208,"num_write":342,"num_write_kb":19130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"44'318","reported_seq":2423,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.672487+0000","last_change":"2024-03-27T06:18:40.829685+0000","last_active":"2024-03-27T06:22:08.672487+0000","last_peered":"2024-03-27T06:22:08.672487+0000","last_clean":"2024-03-27T06:22:08.672487+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:22:08.672487+0000","last_undegraded":"2024-03-27T06:22:08.672487+0000","last_fullsized":"2024-03-27T06:22:08.672487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:06:27.943698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035867099999999999,"stat_sum":{"num_bytes":14543064,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":314,"num_write_kb":14366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"44'382","reported_seq":2487,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.717300+0000","last_change":"2024-03-27T06:18:40.562911+0000","last_active":"2024-03-27T06:22:08.717300+0000","last_peered":"2024-03-27T06:22:08.717300+0000","last_clean":"2024-03-27T06:22:08.717300+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:22:08.717300+0000","last_undegraded":"2024-03-27T06:22:08.717300+0000","last_fullsized":"2024-03-27T06:22:08.717300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002151956,"stat_sum":{"num_bytes":5854740,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":367,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"44'314","reported_seq":2417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.658308+0000","last_change":"2024-03-27T06:18:40.827317+0000","last_active":"2024-03-27T06:22:08.658308+0000","last_peered":"2024-03-27T06:22:08.658308+0000","last_clean":"2024-03-27T06:22:08.658308+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:22:08.658308+0000","last_undegraded":"2024-03-27T06:22:08.658308+0000","last_fullsized":"2024-03-27T06:22:08.658308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:28:40.555644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032490380000000001,"stat_sum":{"num_bytes":14027540,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":18,"num_read_kb":8206,"num_write":307,"num_write_kb":13857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"44'314","reported_seq":2414,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.683280+0000","last_change":"2024-03-27T06:18:40.827238+0000","last_active":"2024-03-27T06:22:08.683280+0000","last_peered":"2024-03-27T06:22:08.683280+0000","last_clean":"2024-03-27T06:22:08.683280+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:22:08.683280+0000","last_undegraded":"2024-03-27T06:22:08.683280+0000","last_fullsized":"2024-03-27T06:22:08.683280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:35:39.617870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003068571,"stat_sum":{"num_bytes":18001138,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":16,"num_read_kb":7383,"num_write":307,"num_write_kb":21016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"44'358","reported_seq":2465,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.672860+0000","last_change":"2024-03-27T06:18:40.829449+0000","last_active":"2024-03-27T06:22:08.672860+0000","last_peered":"2024-03-27T06:22:08.672860+0000","last_clean":"2024-03-27T06:22:08.672860+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:22:08.672860+0000","last_undegraded":"2024-03-27T06:22:08.672860+0000","last_fullsized":"2024-03-27T06:22:08.672860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:47:20.901338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036751610000000001,"stat_sum":{"num_bytes":22534890,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":22,"num_read_kb":12306,"num_write":351,"num_write_kb":22191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"44'359","reported_seq":2469,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.702520+0000","last_change":"2024-03-27T06:18:40.836714+0000","last_active":"2024-03-27T06:22:08.702520+0000","last_peered":"2024-03-27T06:22:08.702520+0000","last_clean":"2024-03-27T06:22:08.702520+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:22:08.702520+0000","last_undegraded":"2024-03-27T06:22:08.702520+0000","last_fullsized":"2024-03-27T06:22:08.702520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044159999999999,"stat_sum":{"num_bytes":14250100,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":24,"num_read_kb":7593,"num_write":351,"num_write_kb":14104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"44'387","reported_seq":2491,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671810+0000","last_change":"2024-03-27T06:18:40.825940+0000","last_active":"2024-03-27T06:22:08.671810+0000","last_peered":"2024-03-27T06:22:08.671810+0000","last_clean":"2024-03-27T06:22:08.671810+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:22:08.671810+0000","last_undegraded":"2024-03-27T06:22:08.671810+0000","last_fullsized":"2024-03-27T06:22:08.671810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:54:24.095203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011795639999999999,"stat_sum":{"num_bytes":10303903,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":19,"num_read_kb":1482,"num_write":377,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"44'380","reported_seq":2494,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.665210+0000","last_change":"2024-03-27T06:18:40.561817+0000","last_active":"2024-03-27T06:22:08.665210+0000","last_peered":"2024-03-27T06:22:08.665210+0000","last_clean":"2024-03-27T06:22:08.665210+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:22:08.665210+0000","last_undegraded":"2024-03-27T06:22:08.665210+0000","last_fullsized":"2024-03-27T06:22:08.665210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016911179999999999,"stat_sum":{"num_bytes":11109721,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":28,"num_read_kb":4123,"num_write":370,"num_write_kb":11044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"44'327","reported_seq":2431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.682425+0000","last_change":"2024-03-27T06:18:40.831829+0000","last_active":"2024-03-27T06:22:08.682425+0000","last_peered":"2024-03-27T06:22:08.682425+0000","last_clean":"2024-03-27T06:22:08.682425+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:22:08.682425+0000","last_undegraded":"2024-03-27T06:22:08.682425+0000","last_fullsized":"2024-03-27T06:22:08.682425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057058689999999997,"stat_sum":{"num_bytes":18559363,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":18,"num_read_kb":4112,"num_write":320,"num_write_kb":18284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"44'18","reported_seq":85,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:35.625793+0000","last_change":"2024-03-27T06:14:02.523274+0000","last_active":"2024-03-27T06:21:35.625793+0000","last_peered":"2024-03-27T06:21:35.625793+0000","last_clean":"2024-03-27T06:21:35.625793+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:21:35.625793+0000","last_undegraded":"2024-03-27T06:21:35.625793+0000","last_fullsized":"2024-03-27T06:21:35.625793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195437,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":19,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'319","reported_seq":2421,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.695864+0000","last_change":"2024-03-27T06:18:40.562829+0000","last_active":"2024-03-27T06:22:08.695864+0000","last_peered":"2024-03-27T06:22:08.695864+0000","last_clean":"2024-03-27T06:22:08.695864+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:22:08.695864+0000","last_undegraded":"2024-03-27T06:22:08.695864+0000","last_fullsized":"2024-03-27T06:22:08.695864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:17:17.674549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018454000000000001,"stat_sum":{"num_bytes":6743271,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":310,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"44'253","reported_seq":327,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.251797+0000","last_change":"2024-03-27T06:14:02.540443+0000","last_active":"2024-03-27T06:21:55.251797+0000","last_peered":"2024-03-27T06:21:55.251797+0000","last_clean":"2024-03-27T06:21:55.251797+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:21:55.251797+0000","last_undegraded":"2024-03-27T06:21:55.251797+0000","last_fullsized":"2024-03-27T06:21:55.251797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":8577,"num_write":252,"num_write_kb":12472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"44'63","reported_seq":133,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250279+0000","last_change":"2024-03-27T06:14:02.534852+0000","last_active":"2024-03-27T06:21:55.250279+0000","last_peered":"2024-03-27T06:21:55.250279+0000","last_clean":"2024-03-27T06:21:55.250279+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:21:55.250279+0000","last_undegraded":"2024-03-27T06:21:55.250279+0000","last_fullsized":"2024-03-27T06:21:55.250279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":65,"num_write_kb":4173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"44'407","reported_seq":2519,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.675998+0000","last_change":"2024-03-27T06:18:40.830513+0000","last_active":"2024-03-27T06:22:08.675998+0000","last_peered":"2024-03-27T06:22:08.675998+0000","last_clean":"2024-03-27T06:22:08.675998+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:22:08.675998+0000","last_undegraded":"2024-03-27T06:22:08.675998+0000","last_fullsized":"2024-03-27T06:22:08.675998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:54.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003103719,"stat_sum":{"num_bytes":18931276,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":28,"num_read_kb":12930,"num_write":396,"num_write_kb":26887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"44'377","reported_seq":443,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:57.232929+0000","last_change":"2024-03-27T06:14:02.537942+0000","last_active":"2024-03-27T06:21:57.232929+0000","last_peered":"2024-03-27T06:21:57.232929+0000","last_clean":"2024-03-27T06:21:57.232929+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:21:57.232929+0000","last_undegraded":"2024-03-27T06:21:57.232929+0000","last_fullsized":"2024-03-27T06:21:57.232929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":374,"num_write_kb":12507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"44'345","reported_seq":2452,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671766+0000","last_change":"2024-03-27T06:18:40.562924+0000","last_active":"2024-03-27T06:22:08.671766+0000","last_peered":"2024-03-27T06:22:08.671766+0000","last_clean":"2024-03-27T06:22:08.671766+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:22:08.671766+0000","last_undegraded":"2024-03-27T06:22:08.671766+0000","last_fullsized":"2024-03-27T06:22:08.671766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:17:24.783114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019163050000000001,"stat_sum":{"num_bytes":19664158,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":13845,"num_write":329,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"44'14","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.299712+0000","last_change":"2024-03-27T06:22:15.299761+0000","last_active":"2024-03-27T06:22:15.299712+0000","last_peered":"2024-03-27T06:22:15.299712+0000","last_clean":"2024-03-27T06:22:15.299712+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:22:15.299712+0000","last_undegraded":"2024-03-27T06:22:15.299712+0000","last_fullsized":"2024-03-27T06:22:15.299712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'14","last_scrub_stamp":"2024-03-27T06:22:15.299611+0000","last_deep_scrub":"44'14","last_deep_scrub_stamp":"2024-03-27T06:22:15.299611+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.299611+0000","objects_scrubbed":4,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:20:07.902308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5952,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"44'361","reported_seq":2472,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.717173+0000","last_change":"2024-03-27T06:18:40.563693+0000","last_active":"2024-03-27T06:22:08.717173+0000","last_peered":"2024-03-27T06:22:08.717173+0000","last_clean":"2024-03-27T06:22:08.717173+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:22:08.717173+0000","last_undegraded":"2024-03-27T06:22:08.717173+0000","last_fullsized":"2024-03-27T06:22:08.717173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026075790000000001,"stat_sum":{"num_bytes":22946854,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":25,"num_read_kb":12308,"num_write":351,"num_write_kb":22593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"44'111","reported_seq":186,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.074537+0000","last_change":"2024-03-27T06:22:14.074616+0000","last_active":"2024-03-27T06:22:14.074537+0000","last_peered":"2024-03-27T06:22:14.074537+0000","last_clean":"2024-03-27T06:22:14.074537+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:22:14.074537+0000","last_undegraded":"2024-03-27T06:22:14.074537+0000","last_fullsized":"2024-03-27T06:22:14.074537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'111","last_scrub_stamp":"2024-03-27T06:22:14.074335+0000","last_deep_scrub":"44'111","last_deep_scrub_stamp":"2024-03-27T06:22:14.074335+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.074335+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:24.848318+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":106,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24559,"num_omap_keys":41,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"44'329","reported_seq":2436,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.722275+0000","last_change":"2024-03-27T06:18:40.830168+0000","last_active":"2024-03-27T06:22:08.722275+0000","last_peered":"2024-03-27T06:22:08.722275+0000","last_clean":"2024-03-27T06:22:08.722275+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:22:08.722275+0000","last_undegraded":"2024-03-27T06:22:08.722275+0000","last_fullsized":"2024-03-27T06:22:08.722275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:10:38.571160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003650304,"stat_sum":{"num_bytes":10082281,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":22,"num_read_kb":4115,"num_write":319,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"44'282","reported_seq":348,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.729600+0000","last_change":"2024-03-27T06:14:02.527023+0000","last_active":"2024-03-27T06:22:08.729600+0000","last_peered":"2024-03-27T06:22:08.729600+0000","last_clean":"2024-03-27T06:22:08.729600+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:22:08.729600+0000","last_undegraded":"2024-03-27T06:22:08.729600+0000","last_fullsized":"2024-03-27T06:22:08.729600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:35.627954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":284,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'326","reported_seq":2428,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.715697+0000","last_change":"2024-03-27T06:18:40.835973+0000","last_active":"2024-03-27T06:22:08.715697+0000","last_peered":"2024-03-27T06:22:08.715697+0000","last_clean":"2024-03-27T06:22:08.715697+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:22:08.715697+0000","last_undegraded":"2024-03-27T06:22:08.715697+0000","last_fullsized":"2024-03-27T06:22:08.715697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084315699999999993,"stat_sum":{"num_bytes":18808790,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":315,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"44'133","reported_seq":212,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.280880+0000","last_change":"2024-03-27T06:22:14.280944+0000","last_active":"2024-03-27T06:22:14.280880+0000","last_peered":"2024-03-27T06:22:14.280880+0000","last_clean":"2024-03-27T06:22:14.280880+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:22:14.280880+0000","last_undegraded":"2024-03-27T06:22:14.280880+0000","last_fullsized":"2024-03-27T06:22:14.280880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'133","last_scrub_stamp":"2024-03-27T06:22:14.280778+0000","last_deep_scrub":"44'133","last_deep_scrub_stamp":"2024-03-27T06:22:14.280778+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.280778+0000","objects_scrubbed":7,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:36:18.758700+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":8203,"num_write":123,"num_write_kb":12364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"44'333","reported_seq":2437,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.693105+0000","last_change":"2024-03-27T06:18:40.841177+0000","last_active":"2024-03-27T06:22:08.693105+0000","last_peered":"2024-03-27T06:22:08.693105+0000","last_clean":"2024-03-27T06:22:08.693105+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:22:08.693105+0000","last_undegraded":"2024-03-27T06:22:08.693105+0000","last_fullsized":"2024-03-27T06:22:08.693105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050386370000000003,"stat_sum":{"num_bytes":2468399,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":327,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"44'145","reported_seq":220,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978468+0000","last_change":"2024-03-27T06:14:02.533696+0000","last_active":"2024-03-27T06:22:03.978468+0000","last_peered":"2024-03-27T06:22:03.978468+0000","last_clean":"2024-03-27T06:22:03.978468+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:22:03.978468+0000","last_undegraded":"2024-03-27T06:22:03.978468+0000","last_fullsized":"2024-03-27T06:22:03.978468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":8224,"num_write":147,"num_write_kb":8324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'350","reported_seq":2459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.673505+0000","last_change":"2024-03-27T06:18:40.826892+0000","last_active":"2024-03-27T06:22:08.673505+0000","last_peered":"2024-03-27T06:22:08.673505+0000","last_clean":"2024-03-27T06:22:08.673505+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:22:08.673505+0000","last_undegraded":"2024-03-27T06:22:08.673505+0000","last_fullsized":"2024-03-27T06:22:08.673505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:55:18.011615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002905866,"stat_sum":{"num_bytes":19666163,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":25,"num_read_kb":16403,"num_write":336,"num_write_kb":27577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"44'6","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978037+0000","last_change":"2024-03-27T06:14:02.539775+0000","last_active":"2024-03-27T06:22:03.978037+0000","last_peered":"2024-03-27T06:22:03.978037+0000","last_clean":"2024-03-27T06:22:03.978037+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:22:03.978037+0000","last_undegraded":"2024-03-27T06:22:03.978037+0000","last_fullsized":"2024-03-27T06:22:03.978037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"44'361","reported_seq":2470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.722176+0000","last_change":"2024-03-27T06:18:40.837185+0000","last_active":"2024-03-27T06:22:08.722176+0000","last_peered":"2024-03-27T06:22:08.722176+0000","last_clean":"2024-03-27T06:22:08.722176+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:22:08.722176+0000","last_undegraded":"2024-03-27T06:22:08.722176+0000","last_fullsized":"2024-03-27T06:22:08.722176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010165321999999999,"stat_sum":{"num_bytes":10617237,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":345,"num_write_kb":10547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"44'15","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.732901+0000","last_change":"2024-03-27T06:14:02.541431+0000","last_active":"2024-03-27T06:22:08.732901+0000","last_peered":"2024-03-27T06:22:08.732901+0000","last_clean":"2024-03-27T06:22:08.732901+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:22:08.732901+0000","last_undegraded":"2024-03-27T06:22:08.732901+0000","last_fullsized":"2024-03-27T06:22:08.732901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1215,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":16,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"44'353","reported_seq":2455,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.684643+0000","last_change":"2024-03-27T06:18:40.831916+0000","last_active":"2024-03-27T06:22:08.684643+0000","last_peered":"2024-03-27T06:22:08.684643+0000","last_clean":"2024-03-27T06:22:08.684643+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:22:08.684643+0000","last_undegraded":"2024-03-27T06:22:08.684643+0000","last_fullsized":"2024-03-27T06:22:08.684643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054212890000000001,"stat_sum":{"num_bytes":24812922,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":17,"num_read_kb":8206,"num_write":342,"num_write_kb":26402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"44'278","reported_seq":359,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.250589+0000","last_change":"2024-03-27T06:14:02.523273+0000","last_active":"2024-03-27T06:21:55.250589+0000","last_peered":"2024-03-27T06:21:55.250589+0000","last_clean":"2024-03-27T06:21:55.250589+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:21:55.250589+0000","last_undegraded":"2024-03-27T06:21:55.250589+0000","last_fullsized":"2024-03-27T06:21:55.250589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:34:37.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":283,"num_write_kb":8391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"44'364","reported_seq":2467,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.716233+0000","last_change":"2024-03-27T06:18:40.834701+0000","last_active":"2024-03-27T06:22:08.716233+0000","last_peered":"2024-03-27T06:22:08.716233+0000","last_clean":"2024-03-27T06:22:08.716233+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:22:08.716233+0000","last_undegraded":"2024-03-27T06:22:08.716233+0000","last_fullsized":"2024-03-27T06:22:08.716233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083622499999999999,"stat_sum":{"num_bytes":12063538,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":20,"num_read_kb":12305,"num_write":353,"num_write_kb":20111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"44'29","reported_seq":98,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.241992+0000","last_change":"2024-03-27T06:14:02.537530+0000","last_active":"2024-03-27T06:21:55.241992+0000","last_peered":"2024-03-27T06:21:55.241992+0000","last_clean":"2024-03-27T06:21:55.241992+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:21:55.241992+0000","last_undegraded":"2024-03-27T06:21:55.241992+0000","last_fullsized":"2024-03-27T06:21:55.241992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4103,"num_write":34,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'296","reported_seq":2395,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.686930+0000","last_change":"2024-03-27T06:18:40.563367+0000","last_active":"2024-03-27T06:22:08.686930+0000","last_peered":"2024-03-27T06:22:08.686930+0000","last_clean":"2024-03-27T06:22:08.686930+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:22:08.686930+0000","last_undegraded":"2024-03-27T06:22:08.686930+0000","last_fullsized":"2024-03-27T06:22:08.686930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022904700000000002,"stat_sum":{"num_bytes":11066744,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":12134,"num_write":288,"num_write_kb":15054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"44'28","reported_seq":93,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.601016+0000","last_change":"2024-03-27T06:14:02.516736+0000","last_active":"2024-03-27T06:21:29.601016+0000","last_peered":"2024-03-27T06:21:29.601016+0000","last_clean":"2024-03-27T06:21:29.601016+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:21:29.601016+0000","last_undegraded":"2024-03-27T06:21:29.601016+0000","last_fullsized":"2024-03-27T06:21:29.601016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:38:33.165084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":30,"num_write_kb":4134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"44'336","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.697672+0000","last_change":"2024-03-27T06:18:40.841709+0000","last_active":"2024-03-27T06:22:08.697672+0000","last_peered":"2024-03-27T06:22:08.697672+0000","last_clean":"2024-03-27T06:22:08.697672+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:22:08.697672+0000","last_undegraded":"2024-03-27T06:22:08.697672+0000","last_fullsized":"2024-03-27T06:22:08.697672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050322370000000002,"stat_sum":{"num_bytes":14587209,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":24,"num_read_kb":12307,"num_write":322,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"44'23","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:30.283694+0000","last_change":"2024-03-27T06:14:02.528740+0000","last_active":"2024-03-27T06:21:30.283694+0000","last_peered":"2024-03-27T06:21:30.283694+0000","last_clean":"2024-03-27T06:21:30.283694+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:21:30.283694+0000","last_undegraded":"2024-03-27T06:21:30.283694+0000","last_fullsized":"2024-03-27T06:21:30.283694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":23,"num_write_kb":4120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"44'362","reported_seq":2464,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.691475+0000","last_change":"2024-03-27T06:18:40.840461+0000","last_active":"2024-03-27T06:22:08.691475+0000","last_peered":"2024-03-27T06:22:08.691475+0000","last_clean":"2024-03-27T06:22:08.691475+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:22:08.691475+0000","last_undegraded":"2024-03-27T06:22:08.691475+0000","last_fullsized":"2024-03-27T06:22:08.691475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070052669999999999,"stat_sum":{"num_bytes":11205183,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":352,"num_write_kb":11129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"44'29","reported_seq":101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:58.564457+0000","last_change":"2024-03-27T06:14:02.529124+0000","last_active":"2024-03-27T06:21:58.564457+0000","last_peered":"2024-03-27T06:21:58.564457+0000","last_clean":"2024-03-27T06:21:58.564457+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:21:58.564457+0000","last_undegraded":"2024-03-27T06:21:58.564457+0000","last_fullsized":"2024-03-27T06:21:58.564457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":32,"num_write_kb":4138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"44'330","reported_seq":2435,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.718416+0000","last_change":"2024-03-27T06:18:40.835886+0000","last_active":"2024-03-27T06:22:08.718416+0000","last_peered":"2024-03-27T06:22:08.718416+0000","last_clean":"2024-03-27T06:22:08.718416+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:22:08.718416+0000","last_undegraded":"2024-03-27T06:22:08.718416+0000","last_fullsized":"2024-03-27T06:22:08.718416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086058899999999997,"stat_sum":{"num_bytes":14608246,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":19,"num_read_kb":4113,"num_write":322,"num_write_kb":18530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'119","reported_seq":230,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.301075+0000","last_change":"2024-03-27T06:22:14.301165+0000","last_active":"2024-03-27T06:22:14.301075+0000","last_peered":"2024-03-27T06:22:14.301075+0000","last_clean":"2024-03-27T06:22:14.301075+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:22:14.301075+0000","last_undegraded":"2024-03-27T06:22:14.301075+0000","last_fullsized":"2024-03-27T06:22:14.301075+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'119","last_scrub_stamp":"2024-03-27T06:22:14.300945+0000","last_deep_scrub":"44'119","last_deep_scrub_stamp":"2024-03-27T06:22:14.300945+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.300945+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:41:17.101796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'84","reported_seq":153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981080+0000","last_change":"2024-03-27T06:14:02.538341+0000","last_active":"2024-03-27T06:22:03.981080+0000","last_peered":"2024-03-27T06:22:03.981080+0000","last_clean":"2024-03-27T06:22:03.981080+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:22:03.981080+0000","last_undegraded":"2024-03-27T06:22:03.981080+0000","last_fullsized":"2024-03-27T06:22:03.981080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":86,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'329","reported_seq":2428,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.666125+0000","last_change":"2024-03-27T06:18:40.563107+0000","last_active":"2024-03-27T06:22:08.666125+0000","last_peered":"2024-03-27T06:22:08.666125+0000","last_clean":"2024-03-27T06:22:08.666125+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:22:08.666125+0000","last_undegraded":"2024-03-27T06:22:08.666125+0000","last_fullsized":"2024-03-27T06:22:08.666125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:37:09.221434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002005691,"stat_sum":{"num_bytes":10118185,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":14,"num_read_kb":4109,"num_write":319,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"44'229","reported_seq":302,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.732434+0000","last_change":"2024-03-27T06:14:02.529400+0000","last_active":"2024-03-27T06:22:08.732434+0000","last_peered":"2024-03-27T06:22:08.732434+0000","last_clean":"2024-03-27T06:22:08.732434+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:22:08.732434+0000","last_undegraded":"2024-03-27T06:22:08.732434+0000","last_fullsized":"2024-03-27T06:22:08.732434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":8210,"num_write":227,"num_write_kb":12456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'331","reported_seq":2426,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.660633+0000","last_change":"2024-03-27T06:18:40.825563+0000","last_active":"2024-03-27T06:22:08.660633+0000","last_peered":"2024-03-27T06:22:08.660633+0000","last_clean":"2024-03-27T06:22:08.660633+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:22:08.660633+0000","last_undegraded":"2024-03-27T06:22:08.660633+0000","last_fullsized":"2024-03-27T06:22:08.660633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:48:12.564802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001779484,"stat_sum":{"num_bytes":10043455,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":324,"num_write_kb":9981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"44'91","reported_seq":242,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:55.249897+0000","last_change":"2024-03-27T06:14:02.527037+0000","last_active":"2024-03-27T06:21:55.249897+0000","last_peered":"2024-03-27T06:21:55.249897+0000","last_clean":"2024-03-27T06:21:55.249897+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:21:55.249897+0000","last_undegraded":"2024-03-27T06:21:55.249897+0000","last_fullsized":"2024-03-27T06:21:55.249897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534287,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":595,"num_write":90,"num_write_kb":602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"44'340","reported_seq":2443,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.674405+0000","last_change":"2024-03-27T06:18:40.830316+0000","last_active":"2024-03-27T06:22:08.674405+0000","last_peered":"2024-03-27T06:22:08.674405+0000","last_clean":"2024-03-27T06:22:08.674405+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:22:08.674405+0000","last_undegraded":"2024-03-27T06:22:08.674405+0000","last_fullsized":"2024-03-27T06:22:08.674405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:36:51.327239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031179929999999999,"stat_sum":{"num_bytes":2779321,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":332,"num_write_kb":2894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"44'41","reported_seq":107,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978136+0000","last_change":"2024-03-27T06:14:02.529778+0000","last_active":"2024-03-27T06:22:03.978136+0000","last_peered":"2024-03-27T06:22:03.978136+0000","last_clean":"2024-03-27T06:22:03.978136+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:22:03.978136+0000","last_undegraded":"2024-03-27T06:22:03.978136+0000","last_fullsized":"2024-03-27T06:22:03.978136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":36,"num_write_kb":8227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"44'353","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.665718+0000","last_change":"2024-03-27T06:18:40.826852+0000","last_active":"2024-03-27T06:22:08.665718+0000","last_peered":"2024-03-27T06:22:08.665718+0000","last_clean":"2024-03-27T06:22:08.665718+0000","last_became_active":"2024-03-27T06:14:49.853879+0000","last_became_peered":"2024-03-27T06:14:49.853879+0000","last_unstale":"2024-03-27T06:22:08.665718+0000","last_undegraded":"2024-03-27T06:22:08.665718+0000","last_fullsized":"2024-03-27T06:22:08.665718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:54:30.479770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022092909999999999,"stat_sum":{"num_bytes":5308376,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":346,"num_write_kb":5365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"44'78","reported_seq":141,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:37.150979+0000","last_change":"2024-03-27T06:14:02.518864+0000","last_active":"2024-03-27T06:21:37.150979+0000","last_peered":"2024-03-27T06:21:37.150979+0000","last_clean":"2024-03-27T06:21:37.150979+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:21:37.150979+0000","last_undegraded":"2024-03-27T06:21:37.150979+0000","last_fullsized":"2024-03-27T06:21:37.150979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:00:41.309006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":4150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"44'465","reported_seq":2589,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.428443+0000","last_change":"2024-03-27T06:22:15.428493+0000","last_active":"2024-03-27T06:22:15.428443+0000","last_peered":"2024-03-27T06:22:15.428443+0000","last_clean":"2024-03-27T06:22:15.428443+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:22:15.428443+0000","last_undegraded":"2024-03-27T06:22:15.428443+0000","last_fullsized":"2024-03-27T06:22:15.428443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'465","last_scrub_stamp":"2024-03-27T06:22:15.428328+0000","last_deep_scrub":"44'465","last_deep_scrub_stamp":"2024-03-27T06:22:15.428328+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.428328+0000","objects_scrubbed":180,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:31:44.636833+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0.002460744,"stat_sum":{"num_bytes":18473602,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":23,"num_read_kb":12305,"num_write":340,"num_write_kb":22314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'588","reported_seq":654,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.735036+0000","last_change":"2024-03-27T06:14:02.535725+0000","last_active":"2024-03-27T06:22:08.735036+0000","last_peered":"2024-03-27T06:22:08.735036+0000","last_clean":"2024-03-27T06:22:08.735036+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:22:08.735036+0000","last_undegraded":"2024-03-27T06:22:08.735036+0000","last_fullsized":"2024-03-27T06:22:08.735036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":589,"num_write_kb":8551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"44'340","reported_seq":2440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.671368+0000","last_change":"2024-03-27T06:18:40.827659+0000","last_active":"2024-03-27T06:22:08.671368+0000","last_peered":"2024-03-27T06:22:08.671368+0000","last_clean":"2024-03-27T06:22:08.671368+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:22:08.671368+0000","last_undegraded":"2024-03-27T06:22:08.671368+0000","last_fullsized":"2024-03-27T06:22:08.671368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:05:16.618277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032966190000000002,"stat_sum":{"num_bytes":6064397,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":331,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"44'139","reported_seq":209,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.628474+0000","last_change":"2024-03-27T06:14:02.535329+0000","last_active":"2024-03-27T06:22:08.628474+0000","last_peered":"2024-03-27T06:22:08.628474+0000","last_clean":"2024-03-27T06:22:08.628474+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:22:08.628474+0000","last_undegraded":"2024-03-27T06:22:08.628474+0000","last_fullsized":"2024-03-27T06:22:08.628474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:03:47.572270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":327,"num_write_kb":2013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"44'322","reported_seq":2423,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.669399+0000","last_change":"2024-03-27T06:18:40.562700+0000","last_active":"2024-03-27T06:22:08.669399+0000","last_peered":"2024-03-27T06:22:08.669399+0000","last_clean":"2024-03-27T06:22:08.669399+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:22:08.669399+0000","last_undegraded":"2024-03-27T06:22:08.669399+0000","last_fullsized":"2024-03-27T06:22:08.669399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:57:15.843788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017608820000000001,"stat_sum":{"num_bytes":20225923,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":318,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"43'79","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:29.215810+0000","last_change":"2024-03-27T06:14:02.540645+0000","last_active":"2024-03-27T06:21:29.215810+0000","last_peered":"2024-03-27T06:21:29.215810+0000","last_clean":"2024-03-27T06:21:29.215810+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:21:29.215810+0000","last_undegraded":"2024-03-27T06:21:29.215810+0000","last_fullsized":"2024-03-27T06:21:29.215810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":78,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'357","reported_seq":2476,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.668195+0000","last_change":"2024-03-27T06:18:40.562997+0000","last_active":"2024-03-27T06:22:08.668195+0000","last_peered":"2024-03-27T06:22:08.668195+0000","last_clean":"2024-03-27T06:22:08.668195+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:22:08.668195+0000","last_undegraded":"2024-03-27T06:22:08.668195+0000","last_fullsized":"2024-03-27T06:22:08.668195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:19:49.684960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018869889999999999,"stat_sum":{"num_bytes":32233430,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":35,"num_read_kb":18086,"num_write":345,"num_write_kb":31650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"44'35","reported_seq":110,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:13.054417+0000","last_change":"2024-03-27T06:22:13.054470+0000","last_active":"2024-03-27T06:22:13.054417+0000","last_peered":"2024-03-27T06:22:13.054417+0000","last_clean":"2024-03-27T06:22:13.054417+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:22:13.054417+0000","last_undegraded":"2024-03-27T06:22:13.054417+0000","last_fullsized":"2024-03-27T06:22:13.054417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'35","last_scrub_stamp":"2024-03-27T06:22:13.054293+0000","last_deep_scrub":"44'35","last_deep_scrub_stamp":"2024-03-27T06:22:13.054293+0000","last_clean_scrub_stamp":"2024-03-27T06:22:13.054293+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:29.311582+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":35,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27209,"num_omap_keys":47,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'444","reported_seq":2548,"reported_epoch":44,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-27T06:22:15.715903+0000","last_change":"2024-03-27T06:22:15.541111+0000","last_active":"2024-03-27T06:22:15.715903+0000","last_peered":"2024-03-27T06:22:15.715903+0000","last_clean":"2024-03-27T06:22:15.715903+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:22:15.715903+0000","last_undegraded":"2024-03-27T06:22:15.715903+0000","last_fullsized":"2024-03-27T06:22:15.715903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":149,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081323179999999995,"stat_sum":{"num_bytes":22949808,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":346,"num_write_kb":22595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"44'121","reported_seq":187,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.570131+0000","last_change":"2024-03-27T06:14:02.533899+0000","last_active":"2024-03-27T06:22:08.570131+0000","last_peered":"2024-03-27T06:22:08.570131+0000","last_clean":"2024-03-27T06:22:08.570131+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:22:08.570131+0000","last_undegraded":"2024-03-27T06:22:08.570131+0000","last_fullsized":"2024-03-27T06:22:08.570131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:39:54.657219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":126,"num_write_kb":4511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"44'365","reported_seq":2471,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.668254+0000","last_change":"2024-03-27T06:18:40.562590+0000","last_active":"2024-03-27T06:22:08.668254+0000","last_peered":"2024-03-27T06:22:08.668254+0000","last_clean":"2024-03-27T06:22:08.668254+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:22:08.668254+0000","last_undegraded":"2024-03-27T06:22:08.668254+0000","last_fullsized":"2024-03-27T06:22:08.668254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:13:24.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001751964,"stat_sum":{"num_bytes":19413938,"num_objects":185,"num_object_clones":0,"num_object_copies":370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":21,"num_read_kb":12304,"num_write":357,"num_write_kb":23191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"44'27","reported_seq":92,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.980288+0000","last_change":"2024-03-27T06:14:02.517530+0000","last_active":"2024-03-27T06:22:03.980288+0000","last_peered":"2024-03-27T06:22:03.980288+0000","last_clean":"2024-03-27T06:22:03.980288+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:22:03.980288+0000","last_undegraded":"2024-03-27T06:22:03.980288+0000","last_fullsized":"2024-03-27T06:22:03.980288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:19:52.360957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":4098,"num_write":32,"num_write_kb":4123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'327","reported_seq":2429,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.665398+0000","last_change":"2024-03-27T06:18:40.826508+0000","last_active":"2024-03-27T06:22:08.665398+0000","last_peered":"2024-03-27T06:22:08.665398+0000","last_clean":"2024-03-27T06:22:08.665398+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:22:08.665398+0000","last_undegraded":"2024-03-27T06:22:08.665398+0000","last_fullsized":"2024-03-27T06:22:08.665398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:52:40.745990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001518585,"stat_sum":{"num_bytes":15553910,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":17,"num_read_kb":5089,"num_write":319,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"44'327","reported_seq":396,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:13.559542+0000","last_change":"2024-03-27T06:14:02.535327+0000","last_active":"2024-03-27T06:22:13.559542+0000","last_peered":"2024-03-27T06:22:13.559542+0000","last_clean":"2024-03-27T06:22:13.559542+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:22:13.559542+0000","last_undegraded":"2024-03-27T06:22:13.559542+0000","last_fullsized":"2024-03-27T06:22:13.559542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10200427,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":324,"num_write_kb":10160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'352","reported_seq":2453,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.710888+0000","last_change":"2024-03-27T06:18:40.836916+0000","last_active":"2024-03-27T06:22:08.710888+0000","last_peered":"2024-03-27T06:22:08.710888+0000","last_clean":"2024-03-27T06:22:08.710888+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:22:08.710888+0000","last_undegraded":"2024-03-27T06:22:08.710888+0000","last_fullsized":"2024-03-27T06:22:08.710888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075044719999999999,"stat_sum":{"num_bytes":10744528,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":347,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'322","reported_seq":2433,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.721886+0000","last_change":"2024-03-27T06:18:40.835719+0000","last_active":"2024-03-27T06:22:08.721886+0000","last_peered":"2024-03-27T06:22:08.721886+0000","last_clean":"2024-03-27T06:22:08.721886+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:22:08.721886+0000","last_undegraded":"2024-03-27T06:22:08.721886+0000","last_fullsized":"2024-03-27T06:22:08.721886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086575760000000002,"stat_sum":{"num_bytes":8008268,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4119,"num_write":307,"num_write_kb":12040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"30'22","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:21:35.838697+0000","last_change":"2024-03-27T06:14:02.535054+0000","last_active":"2024-03-27T06:21:35.838697+0000","last_peered":"2024-03-27T06:21:35.838697+0000","last_clean":"2024-03-27T06:21:35.838697+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:21:35.838697+0000","last_undegraded":"2024-03-27T06:21:35.838697+0000","last_fullsized":"2024-03-27T06:21:35.838697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":22,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"44'344","reported_seq":2445,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.697814+0000","last_change":"2024-03-27T06:18:40.835158+0000","last_active":"2024-03-27T06:22:08.697814+0000","last_peered":"2024-03-27T06:22:08.697814+0000","last_clean":"2024-03-27T06:22:08.697814+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:22:08.697814+0000","last_undegraded":"2024-03-27T06:22:08.697814+0000","last_fullsized":"2024-03-27T06:22:08.697814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061971559999999997,"stat_sum":{"num_bytes":27398907,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":15,"num_read_kb":8205,"num_write":334,"num_write_kb":31029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"44'60","reported_seq":139,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.036903+0000","last_change":"2024-03-27T06:22:15.036966+0000","last_active":"2024-03-27T06:22:15.036903+0000","last_peered":"2024-03-27T06:22:15.036903+0000","last_clean":"2024-03-27T06:22:15.036903+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:22:15.036903+0000","last_undegraded":"2024-03-27T06:22:15.036903+0000","last_fullsized":"2024-03-27T06:22:15.036903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'60","last_scrub_stamp":"2024-03-27T06:22:15.036766+0000","last_deep_scrub":"44'60","last_deep_scrub_stamp":"2024-03-27T06:22:15.036766+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.036766+0000","objects_scrubbed":5,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:00:16.842707+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":12312,"num_write":46,"num_write_kb":12316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"44'318","reported_seq":2424,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.672788+0000","last_change":"2024-03-27T06:18:40.829954+0000","last_active":"2024-03-27T06:22:08.672788+0000","last_peered":"2024-03-27T06:22:08.672788+0000","last_clean":"2024-03-27T06:22:08.672788+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:22:08.672788+0000","last_undegraded":"2024-03-27T06:22:08.672788+0000","last_fullsized":"2024-03-27T06:22:08.672788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:15:13.116715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036426230000000002,"stat_sum":{"num_bytes":14422279,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":21,"num_read_kb":4923,"num_write":311,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"44'81","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.978121+0000","last_change":"2024-03-27T06:14:02.523196+0000","last_active":"2024-03-27T06:22:03.978121+0000","last_peered":"2024-03-27T06:22:03.978121+0000","last_clean":"2024-03-27T06:22:03.978121+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:22:03.978121+0000","last_undegraded":"2024-03-27T06:22:03.978121+0000","last_fullsized":"2024-03-27T06:22:03.978121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":75,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"44'347","reported_seq":2447,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:08.678696+0000","last_change":"2024-03-27T06:18:40.830116+0000","last_active":"2024-03-27T06:22:08.678696+0000","last_peered":"2024-03-27T06:22:08.678696+0000","last_clean":"2024-03-27T06:22:08.678696+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:22:08.678696+0000","last_undegraded":"2024-03-27T06:22:08.678696+0000","last_fullsized":"2024-03-27T06:22:08.678696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:02.488928+0000","last_clean_scrub_stamp":"2024-03-27T06:14:02.488928+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:55.091626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036711629999999999,"stat_sum":{"num_bytes":15186793,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":8204,"num_write":343,"num_write_kb":14950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"44'130","reported_seq":208,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:03.981295+0000","last_change":"2024-03-27T06:14:02.534175+0000","last_active":"2024-03-27T06:22:03.981295+0000","last_peered":"2024-03-27T06:22:03.981295+0000","last_clean":"2024-03-27T06:22:03.981295+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:22:03.981295+0000","last_undegraded":"2024-03-27T06:22:03.981295+0000","last_fullsized":"2024-03-27T06:22:03.981295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T06:14:01.477708+0000","last_clean_scrub_stamp":"2024-03-27T06:14:01.477708+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:20:14.651548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":4136,"num_write":131,"num_write_kb":8314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940893603,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":1256,"num_read_kb":394723,"num_write":21174,"num_write_kb":1064721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1712472064,"data_stored":1880247110,"data_compressed":164906076,"data_compressed_allocated":182468608,"data_compressed_original":405446656,"omap_allocated":0,"internal_metadata":0},"log_size":21969,"ondisk_log_size":21969,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182706508,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":453,"num_read_kb":120084,"num_write":4110,"num_write_kb":183953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":97,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":62720,"num_omap_keys":108,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81502208,"data_stored":365413016,"data_compressed":49784698,"data_compressed_allocated":62349312,"data_compressed_original":346415104,"omap_allocated":0,"internal_metadata":0},"log_size":3936,"ondisk_log_size":3936,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170540,"kb_used_data":143524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94201300,"statfs":{"total":96636764160,"available":96462131200,"internally_reserved":0,"allocated":146968576,"data_stored":174700599,"data_compressed":11348325,"data_compressed_allocated":13275136,"data_compressed_original":46198784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196900,"kb_used_data":169884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94174940,"statfs":{"total":96636764160,"available":96435138560,"internally_reserved":0,"allocated":173961216,"data_stored":215859568,"data_compressed":22390241,"data_compressed_allocated":25247744,"data_compressed_original":73641984,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291908,"kb_used_data":242684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94079932,"statfs":{"total":96636764160,"available":96337850368,"internally_reserved":0,"allocated":248508416,"data_stored":304860587,"data_compressed":28268738,"data_compressed_allocated":32043008,"data_compressed_original":96636928,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266772,"kb_used_data":217548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94105068,"statfs":{"total":96636764160,"available":96363589632,"internally_reserved":0,"allocated":222769152,"data_stored":282203639,"data_compressed":27135224,"data_compressed_allocated":30642176,"data_compressed_original":95821824,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574951,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332192,"kb_used_data":264920,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94039648,"statfs":{"total":96636764160,"available":96296599552,"internally_reserved":0,"allocated":271278080,"data_stored":359480675,"data_compressed":39566853,"data_compressed_allocated":45527040,"data_compressed_original":142249984,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":267636,"kb_used_data":222572,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94104204,"statfs":{"total":96636764160,"available":96362704896,"internally_reserved":0,"allocated":227913728,"data_stored":275692096,"data_compressed":26585237,"data_compressed_allocated":30089216,"data_compressed_original":85848064,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246944,"kb_used_data":201880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94124896,"statfs":{"total":96636764160,"available":96383893504,"internally_reserved":0,"allocated":206725120,"data_stored":255586545,"data_compressed":23288424,"data_compressed_allocated":26611712,"data_compressed_original":81715200,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360220,"kb_used_data":292948,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94011620,"statfs":{"total":96636764160,"available":96267898880,"internally_reserved":0,"allocated":299978752,"data_stored":380226993,"data_compressed":36172470,"data_compressed_allocated":41472000,"data_compressed_original":130895872,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":67110807,"data_compressed":8915435,"data_compressed_allocated":11333632,"data_compressed_original":64311296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7794688,"data_stored":37748896,"data_compressed":4858381,"data_compressed_allocated":6283264,"data_compressed_original":36245504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7049216,"data_stored":31172653,"data_compressed":4475552,"data_compressed_allocated":5537792,"data_compressed_original":29675520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19849216,"data_stored":81505448,"data_compressed":11182288,"data_compressed_allocated":14282752,"data_compressed_original":75956224,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":50333888,"data_compressed":7205527,"data_compressed_allocated":8712192,"data_compressed_original":46735360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8495104,"data_stored":42478552,"data_compressed":5618062,"data_compressed_allocated":6979584,"data_compressed_original":40984576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5763072,"data_stored":29361843,"data_compressed":4140996,"data_compressed_allocated":5042176,"data_compressed_original":28651520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":25700929,"data_compressed":3388457,"data_compressed_allocated":4177920,"data_compressed_original":23855104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285327360,"data_stored":312894956,"data_compressed":27257035,"data_compressed_allocated":30138368,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198365184,"data_stored":217026051,"data_compressed":18397674,"data_compressed_allocated":20283392,"data_compressed_original":44896256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220364800,"data_stored":244298213,"data_compressed":22109685,"data_compressed_allocated":24551424,"data_compressed_original":56172544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250929152,"data_stored":277753997,"data_compressed":28384565,"data_compressed_allocated":31244288,"data_compressed_original":66293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209920000,"data_stored":231648521,"data_compressed":19929697,"data_compressed_allocated":21929984,"data_compressed_original":49086464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239448064,"data_stored":261570437,"data_compressed":22618307,"data_compressed_allocated":25018368,"data_compressed_original":55078912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167698432,"data_stored":186276495,"data_compressed":18249245,"data_compressed_allocated":20205568,"data_compressed_original":44990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140419072,"data_stored":148778440,"data_compressed":7959868,"data_compressed_allocated":9097216,"data_compressed_original":22343680,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:22:18.525 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.526 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.526 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.526 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.526 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.526 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.527 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.528 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.529 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.530 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.531 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.532 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.533 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.534 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.535 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.536 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.537 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.538 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.539 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.540 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.540 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.540 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-27T06:14:02.488928+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=2, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.540 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-03-27T06:14:01.477708+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=14, tm_sec=1, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=6, tm_min=22, tm_sec=11, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T06:22:18.540 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-27T06:22:35.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:22:35.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:22:38.541 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T06:22:38.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.757+0000 7f561be00640 1 Processor -- start 2024-03-27T06:22:38.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.758+0000 7f561be00640 1 -- start start 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f561be00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f56141590a0 0x7f5614071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f561be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f56140716b0 0x7f561406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f561be00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561406b630 con 0x7f5614152950 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f561be00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5614152d30 con 0x7f56141590a0 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f561be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f561406b8d0 con 0x7f56140716b0 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f5618a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f56141590a0 0x7f5614071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f5618a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f56141590a0 0x7f5614071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54010/0 (socket says 172.21.15.132:54010) 2024-03-27T06:22:38.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f5619e00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f56140716b0 0x7f561406cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f5618a00640 1 -- 172.21.15.132:0/2273206268 learned_addr learned my addr 172.21.15.132:0/2273206268 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:38.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.759+0000 7f5619400640 1 --1- 172.21.15.132:0/2273206268 >> v1:172.21.15.132:6789/0 conn(0x7f5614152950 0x7f5614073550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:40380/0 (socket says 172.21.15.132:40380) 2024-03-27T06:22:38.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.760+0000 7f5618a00640 1 -- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f56140716b0 msgr2=0x7f561406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.760+0000 7f5618a00640 1 --2- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f56140716b0 0x7f561406cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.760+0000 7f5618a00640 1 -- 172.21.15.132:0/2273206268 >> v1:172.21.15.132:6789/0 conn(0x7f5614152950 legacy=0x7f5614073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.760+0000 7f5618a00640 1 -- 172.21.15.132:0/2273206268 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561406bbb0 con 0x7f56141590a0 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.761+0000 7f5618a00640 1 --2- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56141590a0 0x7f5614071170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5600009870 tx=0x7f560002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c950c0c08410508 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.761+0000 7f5613e00640 1 -- 172.21.15.132:0/2273206268 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600034070 con 0x7f56141590a0 2024-03-27T06:22:38.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.761+0000 7f5613e00640 1 -- 172.21.15.132:0/2273206268 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5600030840 con 0x7f56141590a0 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.761+0000 7f5613e00640 1 -- 172.21.15.132:0/2273206268 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600030c10 con 0x7f56141590a0 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 -- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56141590a0 msgr2=0x7f5614071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 --2- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56141590a0 0x7f5614071170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5600009870 tx=0x7f560002b980 comp rx=0 tx=0).stop 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 -- 172.21.15.132:0/2273206268 shutdown_connections 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 --2- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f56140716b0 0x7f561406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 --2- 172.21.15.132:0/2273206268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56141590a0 0x7f5614071170 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 -- 172.21.15.132:0/2273206268 >> 172.21.15.132:0/2273206268 conn(0x7f56140cc060 msgr2=0x7f56140cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:38.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 -- 172.21.15.132:0/2273206268 shutdown_connections 2024-03-27T06:22:38.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.762+0000 7f561be00640 1 -- 172.21.15.132:0/2273206268 wait complete. 2024-03-27T06:22:38.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.764+0000 7f561be00640 1 Processor -- start 2024-03-27T06:22:38.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.765+0000 7f561be00640 1 -- start start 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 0x7f5614157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56141590a0 0x7f5614157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f561406c450 con 0x7f56141590a0 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f561406c170 con 0x7f56140716b0 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f561be00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f56141594c0 con 0x7f5614152950 2024-03-27T06:22:38.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f5619400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f5619400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54018/0 (socket says 172.21.15.132:54018) 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f5619400640 1 -- 172.21.15.132:0/2081844832 learned_addr learned my addr 172.21.15.132:0/2081844832 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f5619e00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56141590a0 0x7f5614157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.766+0000 7f5618a00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 0x7f5614157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 -- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 msgr2=0x7f5614157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 0x7f5614157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 -- 172.21.15.132:0/2081844832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56141590a0 msgr2=0x7f5614157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56141590a0 0x7f5614157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 -- 172.21.15.132:0/2081844832 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56140b1260 con 0x7f56140716b0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5618a00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 0x7f5614157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5619400640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f560800c460 tx=0x7f560800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608018070 con 0x7f56140716b0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5608007480 con 0x7f56140716b0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.767+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56140c4c40 con 0x7f56140716b0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.768+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56080047e0 con 0x7f56140716b0 2024-03-27T06:22:38.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.768+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56140afd30 con 0x7f56140716b0 2024-03-27T06:22:38.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.769+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5614068840 con 0x7f56140716b0 2024-03-27T06:22:38.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.770+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f560801b070 con 0x7f56140716b0 2024-03-27T06:22:38.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.771+0000 7f5611600640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 0x7f55f406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:38.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.771+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f560808aad0 con 0x7f56140716b0 2024-03-27T06:22:38.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.771+0000 7f5618a00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 0x7f55f406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:38.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.771+0000 7f5618a00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 0x7f55f406e9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5600036000 tx=0x7f560003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:38.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.775+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f56080587a0 con 0x7f56140716b0 2024-03-27T06:22:38.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.968+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 --> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f561406dac0 con 0x7f55f406c530 2024-03-27T06:22:38.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.976+0000 7f5611600640 1 -- 172.21.15.132:0/2081844832 <== mgr.4106 v2:172.21.15.134:6832/2615516388 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257618 (secure 0 0 0) 0x7f561406dac0 con 0x7f55f406c530 2024-03-27T06:22:38.969 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:38.971 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-03-27T06:22:38.972 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.981+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 msgr2=0x7f55f406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.981+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 0x7f55f406e9f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5600036000 tx=0x7f560003b040 comp rx=0 tx=0).stop 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 msgr2=0x7f561414d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f560800c460 tx=0x7f560800c920 comp rx=0 tx=0).stop 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 shutdown_connections 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56141590a0 0x7f5614157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5614152950 0x7f5614157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f55f406c530 0x7f55f406e9f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 --2- 172.21.15.132:0/2081844832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56140716b0 0x7f561414d8e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:38.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 >> 172.21.15.132:0/2081844832 conn(0x7f56140cc060 msgr2=0x7f561406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:38.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 shutdown_connections 2024-03-27T06:22:38.974 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:38.982+0000 7f561be00640 1 -- 172.21.15.132:0/2081844832 wait complete. 2024-03-27T06:22:38.990 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-03-27T06:22:38.486586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1124395553,"num_objects":10759,"num_object_clones":0,"num_object_copies":21518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10759,"num_whiteouts":0,"num_read":1825,"num_read_kb":514907,"num_write":25463,"num_write_kb":1251300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":97,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706809,"num_omap_keys":1217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33065,"ondisk_log_size":33065,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2198528,"kb_used_data":1756160,"kb_used_omap":12,"kb_used_meta":442291,"kb_avail":752776192,"statfs":{"total":773094113280,"available":770842820608,"internally_reserved":0,"allocated":1798307840,"data_stored":2249020850,"data_compressed":214789432,"data_compressed_allocated":244973568,"data_compressed_original":753278976,"omap_allocated":12751,"internal_metadata":452906545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45877,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":46584,"num_omap_keys":80,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002353"},"pg_stats":[{"pgid":"3.3f","version":"44'471","reported_seq":2590,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:28.274113+0000","last_change":"2024-03-27T06:22:28.274174+0000","last_active":"2024-03-27T06:22:28.274113+0000","last_peered":"2024-03-27T06:22:28.274113+0000","last_clean":"2024-03-27T06:22:28.274113+0000","last_became_active":"2024-03-27T06:14:03.555532+0000","last_became_peered":"2024-03-27T06:14:03.555532+0000","last_unstale":"2024-03-27T06:22:28.274113+0000","last_undegraded":"2024-03-27T06:22:28.274113+0000","last_fullsized":"2024-03-27T06:22:28.274113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'471","last_scrub_stamp":"2024-03-27T06:22:28.274021+0000","last_deep_scrub":"44'471","last_deep_scrub_stamp":"2024-03-27T06:22:28.274021+0000","last_clean_scrub_stamp":"2024-03-27T06:22:28.274021+0000","objects_scrubbed":173,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:50:12.717216+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0.0048207270000000003,"stat_sum":{"num_bytes":26656138,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":18,"num_read_kb":8207,"num_write":331,"num_write_kb":26201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"44'435","reported_seq":2552,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:22.195499+0000","last_change":"2024-03-27T06:22:22.195499+0000","last_active":"2024-03-27T06:22:22.195499+0000","last_peered":"2024-03-27T06:22:22.195499+0000","last_clean":"2024-03-27T06:22:22.195499+0000","last_became_active":"2024-03-27T06:14:03.524031+0000","last_became_peered":"2024-03-27T06:14:03.524031+0000","last_unstale":"2024-03-27T06:22:22.195499+0000","last_undegraded":"2024-03-27T06:22:22.195499+0000","last_fullsized":"2024-03-27T06:22:22.195499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'435","last_scrub_stamp":"2024-03-27T06:22:22.195342+0000","last_deep_scrub":"44'435","last_deep_scrub_stamp":"2024-03-27T06:22:22.195342+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.195342+0000","objects_scrubbed":174,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:19:16.424497+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0.0037350370000000001,"stat_sum":{"num_bytes":5169603,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":315,"num_write_kb":5186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"44'432","reported_seq":2551,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:25.202540+0000","last_change":"2024-03-27T06:22:25.202540+0000","last_active":"2024-03-27T06:22:25.202540+0000","last_peered":"2024-03-27T06:22:25.202540+0000","last_clean":"2024-03-27T06:22:25.202540+0000","last_became_active":"2024-03-27T06:14:03.528709+0000","last_became_peered":"2024-03-27T06:14:03.528709+0000","last_unstale":"2024-03-27T06:22:25.202540+0000","last_undegraded":"2024-03-27T06:22:25.202540+0000","last_fullsized":"2024-03-27T06:22:25.202540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'432","last_scrub_stamp":"2024-03-27T06:22:25.202418+0000","last_deep_scrub":"44'432","last_deep_scrub_stamp":"2024-03-27T06:22:25.202418+0000","last_clean_scrub_stamp":"2024-03-27T06:22:25.202418+0000","objects_scrubbed":155,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:27:29.960995+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0.0036092289999999998,"stat_sum":{"num_bytes":13816404,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":21,"num_read_kb":8211,"num_write":321,"num_write_kb":21844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"44'412","reported_seq":2526,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.368762+0000","last_change":"2024-03-27T06:22:17.368855+0000","last_active":"2024-03-27T06:22:17.368762+0000","last_peered":"2024-03-27T06:22:17.368762+0000","last_clean":"2024-03-27T06:22:17.368762+0000","last_became_active":"2024-03-27T06:14:03.529673+0000","last_became_peered":"2024-03-27T06:14:03.529673+0000","last_unstale":"2024-03-27T06:22:17.368762+0000","last_undegraded":"2024-03-27T06:22:17.368762+0000","last_fullsized":"2024-03-27T06:22:17.368762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'412","last_scrub_stamp":"2024-03-27T06:22:17.368644+0000","last_deep_scrub":"44'412","last_deep_scrub_stamp":"2024-03-27T06:22:17.368644+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.368644+0000","objects_scrubbed":173,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:04:13.651832+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0.002773321,"stat_sum":{"num_bytes":19633039,"num_objects":173,"num_object_clones":0,"num_object_copies":346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":13,"num_read_kb":8202,"num_write":307,"num_write_kb":19330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"44'449","reported_seq":2568,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.185195+0000","last_change":"2024-03-27T06:22:20.185236+0000","last_active":"2024-03-27T06:22:20.185195+0000","last_peered":"2024-03-27T06:22:20.185195+0000","last_clean":"2024-03-27T06:22:20.185195+0000","last_became_active":"2024-03-27T06:14:03.546950+0000","last_became_peered":"2024-03-27T06:14:03.546950+0000","last_unstale":"2024-03-27T06:22:20.185195+0000","last_undegraded":"2024-03-27T06:22:20.185195+0000","last_fullsized":"2024-03-27T06:22:20.185195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'449","last_scrub_stamp":"2024-03-27T06:22:20.185077+0000","last_deep_scrub":"44'449","last_deep_scrub_stamp":"2024-03-27T06:22:20.185077+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.185077+0000","objects_scrubbed":153,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:50:21.062586+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0.0071357900000000004,"stat_sum":{"num_bytes":5850441,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":350,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"44'412","reported_seq":2531,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.278446+0000","last_change":"2024-03-27T06:22:18.278537+0000","last_active":"2024-03-27T06:22:18.278446+0000","last_peered":"2024-03-27T06:22:18.278446+0000","last_clean":"2024-03-27T06:22:18.278446+0000","last_became_active":"2024-03-27T06:14:03.531169+0000","last_became_peered":"2024-03-27T06:14:03.531169+0000","last_unstale":"2024-03-27T06:22:18.278446+0000","last_undegraded":"2024-03-27T06:22:18.278446+0000","last_fullsized":"2024-03-27T06:22:18.278446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'412","last_scrub_stamp":"2024-03-27T06:22:18.278329+0000","last_deep_scrub":"44'412","last_deep_scrub_stamp":"2024-03-27T06:22:18.278329+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.278329+0000","objects_scrubbed":146,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:57:31.354014+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0.002466584,"stat_sum":{"num_bytes":14040535,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":20,"num_read_kb":3571,"num_write":321,"num_write_kb":13825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"44'450","reported_seq":2568,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.204585+0000","last_change":"2024-03-27T06:22:19.204615+0000","last_active":"2024-03-27T06:22:19.204585+0000","last_peered":"2024-03-27T06:22:19.204585+0000","last_clean":"2024-03-27T06:22:19.204585+0000","last_became_active":"2024-03-27T06:14:03.544262+0000","last_became_peered":"2024-03-27T06:14:03.544262+0000","last_unstale":"2024-03-27T06:22:19.204585+0000","last_undegraded":"2024-03-27T06:22:19.204585+0000","last_fullsized":"2024-03-27T06:22:19.204585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'450","last_scrub_stamp":"2024-03-27T06:22:19.204517+0000","last_deep_scrub":"44'450","last_deep_scrub_stamp":"2024-03-27T06:22:19.204517+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.204517+0000","objects_scrubbed":153,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:23:11.678870+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0.0056884500000000003,"stat_sum":{"num_bytes":24259091,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":18,"num_read_kb":4113,"num_write":354,"num_write_kb":23807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"44'481","reported_seq":2594,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:30.325730+0000","last_change":"2024-03-27T06:22:30.325790+0000","last_active":"2024-03-27T06:22:30.325730+0000","last_peered":"2024-03-27T06:22:30.325730+0000","last_clean":"2024-03-27T06:22:30.325730+0000","last_became_active":"2024-03-27T06:14:03.541869+0000","last_became_peered":"2024-03-27T06:14:03.541869+0000","last_unstale":"2024-03-27T06:22:30.325730+0000","last_undegraded":"2024-03-27T06:22:30.325730+0000","last_fullsized":"2024-03-27T06:22:30.325730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'481","last_scrub_stamp":"2024-03-27T06:22:30.325634+0000","last_deep_scrub":"44'481","last_deep_scrub_stamp":"2024-03-27T06:22:30.325634+0000","last_clean_scrub_stamp":"2024-03-27T06:22:30.325634+0000","objects_scrubbed":196,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:09:47.342409+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0.0033930689999999999,"stat_sum":{"num_bytes":15911530,"num_objects":196,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":12,"num_read_kb":4107,"num_write":326,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"44'433","reported_seq":2552,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.462859+0000","last_change":"2024-03-27T06:22:18.462896+0000","last_active":"2024-03-27T06:22:18.462859+0000","last_peered":"2024-03-27T06:22:18.462859+0000","last_clean":"2024-03-27T06:22:18.462859+0000","last_became_active":"2024-03-27T06:14:03.540375+0000","last_became_peered":"2024-03-27T06:14:03.540375+0000","last_unstale":"2024-03-27T06:22:18.462859+0000","last_undegraded":"2024-03-27T06:22:18.462859+0000","last_fullsized":"2024-03-27T06:22:18.462859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'433","last_scrub_stamp":"2024-03-27T06:22:18.462788+0000","last_deep_scrub":"44'433","last_deep_scrub_stamp":"2024-03-27T06:22:18.462788+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.462788+0000","objects_scrubbed":171,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:26:57.757805+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0.0054729009999999996,"stat_sum":{"num_bytes":6593503,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":20,"num_read_kb":4119,"num_write":318,"num_write_kb":10665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"44'435","reported_seq":2554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:27.259446+0000","last_change":"2024-03-27T06:22:27.259486+0000","last_active":"2024-03-27T06:22:27.259446+0000","last_peered":"2024-03-27T06:22:27.259446+0000","last_clean":"2024-03-27T06:22:27.259446+0000","last_became_active":"2024-03-27T06:14:03.553389+0000","last_became_peered":"2024-03-27T06:14:03.553389+0000","last_unstale":"2024-03-27T06:22:27.259446+0000","last_undegraded":"2024-03-27T06:22:27.259446+0000","last_fullsized":"2024-03-27T06:22:27.259446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'435","last_scrub_stamp":"2024-03-27T06:22:27.259377+0000","last_deep_scrub":"44'435","last_deep_scrub_stamp":"2024-03-27T06:22:27.259377+0000","last_clean_scrub_stamp":"2024-03-27T06:22:27.259377+0000","objects_scrubbed":141,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:49:30.212033+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0.0068694350000000001,"stat_sum":{"num_bytes":18722866,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":20,"num_read_kb":9776,"num_write":307,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"44'387","reported_seq":2507,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.169961+0000","last_change":"2024-03-27T06:22:23.170007+0000","last_active":"2024-03-27T06:22:23.169961+0000","last_peered":"2024-03-27T06:22:23.169961+0000","last_clean":"2024-03-27T06:22:23.169961+0000","last_became_active":"2024-03-27T06:14:03.551502+0000","last_became_peered":"2024-03-27T06:14:03.551502+0000","last_unstale":"2024-03-27T06:22:23.169961+0000","last_undegraded":"2024-03-27T06:22:23.169961+0000","last_fullsized":"2024-03-27T06:22:23.169961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'387","last_scrub_stamp":"2024-03-27T06:22:23.169885+0000","last_deep_scrub":"44'387","last_deep_scrub_stamp":"2024-03-27T06:22:23.169885+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.169885+0000","objects_scrubbed":131,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:00:06.820134+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0.0070900370000000004,"stat_sum":{"num_bytes":18171100,"num_objects":131,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":21,"num_read_kb":8210,"num_write":297,"num_write_kb":17901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"44'394","reported_seq":2503,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:26.193877+0000","last_change":"2024-03-27T06:22:26.193908+0000","last_active":"2024-03-27T06:22:26.193877+0000","last_peered":"2024-03-27T06:22:26.193877+0000","last_clean":"2024-03-27T06:22:26.193877+0000","last_became_active":"2024-03-27T06:14:03.552925+0000","last_became_peered":"2024-03-27T06:14:03.552925+0000","last_unstale":"2024-03-27T06:22:26.193877+0000","last_undegraded":"2024-03-27T06:22:26.193877+0000","last_fullsized":"2024-03-27T06:22:26.193877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'394","last_scrub_stamp":"2024-03-27T06:22:26.193794+0000","last_deep_scrub":"44'394","last_deep_scrub_stamp":"2024-03-27T06:22:26.193794+0000","last_clean_scrub_stamp":"2024-03-27T06:22:26.193794+0000","objects_scrubbed":156,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:11:36.680259+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0.0072900689999999997,"stat_sum":{"num_bytes":8281717,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":296,"num_write_kb":12341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"44'386","reported_seq":2505,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:25.806651+0000","last_change":"2024-03-27T06:22:25.806709+0000","last_active":"2024-03-27T06:22:25.806651+0000","last_peered":"2024-03-27T06:22:25.806651+0000","last_clean":"2024-03-27T06:22:25.806651+0000","last_became_active":"2024-03-27T06:14:03.552343+0000","last_became_peered":"2024-03-27T06:14:03.552343+0000","last_unstale":"2024-03-27T06:22:25.806651+0000","last_undegraded":"2024-03-27T06:22:25.806651+0000","last_fullsized":"2024-03-27T06:22:25.806651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'386","last_scrub_stamp":"2024-03-27T06:22:25.806550+0000","last_deep_scrub":"44'386","last_deep_scrub_stamp":"2024-03-27T06:22:25.806550+0000","last_clean_scrub_stamp":"2024-03-27T06:22:25.806550+0000","objects_scrubbed":141,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:08:00.994523+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0.0029554939999999999,"stat_sum":{"num_bytes":19210972,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":21,"num_read_kb":4116,"num_write":287,"num_write_kb":27094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"44'446","reported_seq":2566,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.800622+0000","last_change":"2024-03-27T06:22:21.800655+0000","last_active":"2024-03-27T06:22:21.800622+0000","last_peered":"2024-03-27T06:22:21.800622+0000","last_clean":"2024-03-27T06:22:21.800622+0000","last_became_active":"2024-03-27T06:14:03.516479+0000","last_became_peered":"2024-03-27T06:14:03.516479+0000","last_unstale":"2024-03-27T06:22:21.800622+0000","last_undegraded":"2024-03-27T06:22:21.800622+0000","last_fullsized":"2024-03-27T06:22:21.800622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'446","last_scrub_stamp":"2024-03-27T06:22:21.800552+0000","last_deep_scrub":"44'446","last_deep_scrub_stamp":"2024-03-27T06:22:21.800552+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.800552+0000","objects_scrubbed":164,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:59:21.936029+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0.003176613,"stat_sum":{"num_bytes":7395932,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":20,"num_read_kb":4114,"num_write":334,"num_write_kb":11493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"44'505","reported_seq":2626,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:32.314373+0000","last_change":"2024-03-27T06:22:32.314433+0000","last_active":"2024-03-27T06:22:32.314373+0000","last_peered":"2024-03-27T06:22:32.314373+0000","last_clean":"2024-03-27T06:22:32.314373+0000","last_became_active":"2024-03-27T06:14:03.547213+0000","last_became_peered":"2024-03-27T06:14:03.547213+0000","last_unstale":"2024-03-27T06:22:32.314373+0000","last_undegraded":"2024-03-27T06:22:32.314373+0000","last_fullsized":"2024-03-27T06:22:32.314373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'505","last_scrub_stamp":"2024-03-27T06:22:32.314275+0000","last_deep_scrub":"44'505","last_deep_scrub_stamp":"2024-03-27T06:22:32.314275+0000","last_clean_scrub_stamp":"2024-03-27T06:22:32.314275+0000","objects_scrubbed":193,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:08:38.930992+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0.0036973140000000002,"stat_sum":{"num_bytes":23111157,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":349,"num_write_kb":22756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"44'490","reported_seq":2610,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:29.722321+0000","last_change":"2024-03-27T06:22:29.722379+0000","last_active":"2024-03-27T06:22:29.722321+0000","last_peered":"2024-03-27T06:22:29.722321+0000","last_clean":"2024-03-27T06:22:29.722321+0000","last_became_active":"2024-03-27T06:14:03.539102+0000","last_became_peered":"2024-03-27T06:14:03.539102+0000","last_unstale":"2024-03-27T06:22:29.722321+0000","last_undegraded":"2024-03-27T06:22:29.722321+0000","last_fullsized":"2024-03-27T06:22:29.722321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'490","last_scrub_stamp":"2024-03-27T06:22:29.722252+0000","last_deep_scrub":"44'490","last_deep_scrub_stamp":"2024-03-27T06:22:29.722252+0000","last_clean_scrub_stamp":"2024-03-27T06:22:29.722252+0000","objects_scrubbed":169,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:56:09.150713+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.001289739,"stat_sum":{"num_bytes":12029261,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":347,"num_write_kb":15208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"44'514","reported_seq":2635,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:35.658907+0000","last_change":"2024-03-27T06:22:35.658971+0000","last_active":"2024-03-27T06:22:35.658907+0000","last_peered":"2024-03-27T06:22:35.658907+0000","last_clean":"2024-03-27T06:22:35.658907+0000","last_became_active":"2024-03-27T06:14:03.557507+0000","last_became_peered":"2024-03-27T06:14:03.557507+0000","last_unstale":"2024-03-27T06:22:35.658907+0000","last_undegraded":"2024-03-27T06:22:35.658907+0000","last_fullsized":"2024-03-27T06:22:35.658907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'514","last_scrub_stamp":"2024-03-27T06:22:35.658776+0000","last_deep_scrub":"44'514","last_deep_scrub_stamp":"2024-03-27T06:22:35.658776+0000","last_clean_scrub_stamp":"2024-03-27T06:22:35.658776+0000","objects_scrubbed":167,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:47:51.861192+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0.0089932469999999994,"stat_sum":{"num_bytes":10346712,"num_objects":167,"num_object_clones":0,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":22,"num_read_kb":4117,"num_write":335,"num_write_kb":14376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"44'410","reported_seq":2535,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.548773+0000","last_change":"2024-03-27T06:22:23.548841+0000","last_active":"2024-03-27T06:22:23.548773+0000","last_peered":"2024-03-27T06:22:23.548773+0000","last_clean":"2024-03-27T06:22:23.548773+0000","last_became_active":"2024-03-27T06:14:03.530604+0000","last_became_peered":"2024-03-27T06:14:03.530604+0000","last_unstale":"2024-03-27T06:22:23.548773+0000","last_undegraded":"2024-03-27T06:22:23.548773+0000","last_fullsized":"2024-03-27T06:22:23.548773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'410","last_scrub_stamp":"2024-03-27T06:22:23.548677+0000","last_deep_scrub":"44'410","last_deep_scrub_stamp":"2024-03-27T06:22:23.548677+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.548677+0000","objects_scrubbed":148,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:34:30.796436+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0.0010652260000000001,"stat_sum":{"num_bytes":8103937,"num_objects":148,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":302,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"44'427","reported_seq":2549,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.645221+0000","last_change":"2024-03-27T06:22:18.645267+0000","last_active":"2024-03-27T06:22:18.645221+0000","last_peered":"2024-03-27T06:22:18.645221+0000","last_clean":"2024-03-27T06:22:18.645221+0000","last_became_active":"2024-03-27T06:14:03.539958+0000","last_became_peered":"2024-03-27T06:14:03.539958+0000","last_unstale":"2024-03-27T06:22:18.645221+0000","last_undegraded":"2024-03-27T06:22:18.645221+0000","last_fullsized":"2024-03-27T06:22:18.645221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'427","last_scrub_stamp":"2024-03-27T06:22:18.645123+0000","last_deep_scrub":"44'427","last_deep_scrub_stamp":"2024-03-27T06:22:18.645123+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.645123+0000","objects_scrubbed":157,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:19:12.874279+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0.0075917839999999999,"stat_sum":{"num_bytes":14133748,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":23,"num_read_kb":12737,"num_write":315,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"44'457","reported_seq":2580,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:22.193643+0000","last_change":"2024-03-27T06:22:22.193673+0000","last_active":"2024-03-27T06:22:22.193643+0000","last_peered":"2024-03-27T06:22:22.193643+0000","last_clean":"2024-03-27T06:22:22.193643+0000","last_became_active":"2024-03-27T06:14:03.548313+0000","last_became_peered":"2024-03-27T06:14:03.548313+0000","last_unstale":"2024-03-27T06:22:22.193643+0000","last_undegraded":"2024-03-27T06:22:22.193643+0000","last_fullsized":"2024-03-27T06:22:22.193643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'457","last_scrub_stamp":"2024-03-27T06:22:22.193575+0000","last_deep_scrub":"44'457","last_deep_scrub_stamp":"2024-03-27T06:22:22.193575+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.193575+0000","objects_scrubbed":156,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:08.340582+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.0072459589999999997,"stat_sum":{"num_bytes":23160385,"num_objects":156,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":24,"num_read_kb":12308,"num_write":342,"num_write_kb":30986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"44'455","reported_seq":2581,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:22.066836+0000","last_change":"2024-03-27T06:22:22.066911+0000","last_active":"2024-03-27T06:22:22.066836+0000","last_peered":"2024-03-27T06:22:22.066836+0000","last_clean":"2024-03-27T06:22:22.066836+0000","last_became_active":"2024-03-27T06:14:03.528355+0000","last_became_peered":"2024-03-27T06:14:03.528355+0000","last_unstale":"2024-03-27T06:22:22.066836+0000","last_undegraded":"2024-03-27T06:22:22.066836+0000","last_fullsized":"2024-03-27T06:22:22.066836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'455","last_scrub_stamp":"2024-03-27T06:22:22.066755+0000","last_deep_scrub":"44'455","last_deep_scrub_stamp":"2024-03-27T06:22:22.066755+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.066755+0000","objects_scrubbed":171,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:08:06.660198+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0.001666102,"stat_sum":{"num_bytes":14662751,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":26,"num_read_kb":4119,"num_write":336,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"44'427","reported_seq":2544,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.392025+0000","last_change":"2024-03-27T06:22:20.392050+0000","last_active":"2024-03-27T06:22:20.392025+0000","last_peered":"2024-03-27T06:22:20.392025+0000","last_clean":"2024-03-27T06:22:20.392025+0000","last_became_active":"2024-03-27T06:14:03.558105+0000","last_became_peered":"2024-03-27T06:14:03.558105+0000","last_unstale":"2024-03-27T06:22:20.392025+0000","last_undegraded":"2024-03-27T06:22:20.392025+0000","last_fullsized":"2024-03-27T06:22:20.392025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'427","last_scrub_stamp":"2024-03-27T06:22:20.391957+0000","last_deep_scrub":"44'427","last_deep_scrub_stamp":"2024-03-27T06:22:20.391957+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.391957+0000","objects_scrubbed":152,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:40:04.598342+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0.0024688459999999998,"stat_sum":{"num_bytes":9427787,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":326,"num_write_kb":13475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"44'432","reported_seq":2560,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:26.203774+0000","last_change":"2024-03-27T06:22:26.203816+0000","last_active":"2024-03-27T06:22:26.203774+0000","last_peered":"2024-03-27T06:22:26.203774+0000","last_clean":"2024-03-27T06:22:26.203774+0000","last_became_active":"2024-03-27T06:14:03.529364+0000","last_became_peered":"2024-03-27T06:14:03.529364+0000","last_unstale":"2024-03-27T06:22:26.203774+0000","last_undegraded":"2024-03-27T06:22:26.203774+0000","last_fullsized":"2024-03-27T06:22:26.203774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'432","last_scrub_stamp":"2024-03-27T06:22:26.203703+0000","last_deep_scrub":"44'432","last_deep_scrub_stamp":"2024-03-27T06:22:26.203703+0000","last_clean_scrub_stamp":"2024-03-27T06:22:26.203703+0000","objects_scrubbed":155,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:18:45.414022+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0.0039259519999999999,"stat_sum":{"num_bytes":16419938,"num_objects":155,"num_object_clones":0,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":29,"num_read_kb":18059,"num_write":326,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"44'467","reported_seq":2585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:31.841552+0000","last_change":"2024-03-27T06:22:31.841646+0000","last_active":"2024-03-27T06:22:31.841552+0000","last_peered":"2024-03-27T06:22:31.841552+0000","last_clean":"2024-03-27T06:22:31.841552+0000","last_became_active":"2024-03-27T06:14:03.541154+0000","last_became_peered":"2024-03-27T06:14:03.541154+0000","last_unstale":"2024-03-27T06:22:31.841552+0000","last_undegraded":"2024-03-27T06:22:31.841552+0000","last_fullsized":"2024-03-27T06:22:31.841552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'467","last_scrub_stamp":"2024-03-27T06:22:31.841368+0000","last_deep_scrub":"44'467","last_deep_scrub_stamp":"2024-03-27T06:22:31.841368+0000","last_clean_scrub_stamp":"2024-03-27T06:22:31.841368+0000","objects_scrubbed":168,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:30:49.149234+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0.001470573,"stat_sum":{"num_bytes":19403206,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":18,"num_read_kb":8208,"num_write":342,"num_write_kb":19130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"44'443","reported_seq":2563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:29.232687+0000","last_change":"2024-03-27T06:22:29.232716+0000","last_active":"2024-03-27T06:22:29.232687+0000","last_peered":"2024-03-27T06:22:29.232687+0000","last_clean":"2024-03-27T06:22:29.232687+0000","last_became_active":"2024-03-27T06:14:03.537430+0000","last_became_peered":"2024-03-27T06:14:03.537430+0000","last_unstale":"2024-03-27T06:22:29.232687+0000","last_undegraded":"2024-03-27T06:22:29.232687+0000","last_fullsized":"2024-03-27T06:22:29.232687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'443","last_scrub_stamp":"2024-03-27T06:22:29.232621+0000","last_deep_scrub":"44'443","last_deep_scrub_stamp":"2024-03-27T06:22:29.232621+0000","last_clean_scrub_stamp":"2024-03-27T06:22:29.232621+0000","objects_scrubbed":154,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:24:53.237764+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0.0035867099999999999,"stat_sum":{"num_bytes":14543064,"num_objects":154,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":20,"num_read_kb":4115,"num_write":314,"num_write_kb":14366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"44'499","reported_seq":2619,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.291122+0000","last_change":"2024-03-27T06:22:19.291172+0000","last_active":"2024-03-27T06:22:19.291122+0000","last_peered":"2024-03-27T06:22:19.291122+0000","last_clean":"2024-03-27T06:22:19.291122+0000","last_became_active":"2024-03-27T06:14:03.543441+0000","last_became_peered":"2024-03-27T06:14:03.543441+0000","last_unstale":"2024-03-27T06:22:19.291122+0000","last_undegraded":"2024-03-27T06:22:19.291122+0000","last_fullsized":"2024-03-27T06:22:19.291122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'499","last_scrub_stamp":"2024-03-27T06:22:19.291044+0000","last_deep_scrub":"44'499","last_deep_scrub_stamp":"2024-03-27T06:22:19.291044+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.291044+0000","objects_scrubbed":179,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:55:24.969427+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0.002151956,"stat_sum":{"num_bytes":5854740,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":367,"num_write_kb":5921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"44'416","reported_seq":2535,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.838962+0000","last_change":"2024-03-27T06:22:20.838996+0000","last_active":"2024-03-27T06:22:20.838962+0000","last_peered":"2024-03-27T06:22:20.838962+0000","last_clean":"2024-03-27T06:22:20.838962+0000","last_became_active":"2024-03-27T06:14:03.515939+0000","last_became_peered":"2024-03-27T06:14:03.515939+0000","last_unstale":"2024-03-27T06:22:20.838962+0000","last_undegraded":"2024-03-27T06:22:20.838962+0000","last_fullsized":"2024-03-27T06:22:20.838962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'416","last_scrub_stamp":"2024-03-27T06:22:20.838887+0000","last_deep_scrub":"44'416","last_deep_scrub_stamp":"2024-03-27T06:22:20.838887+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.838887+0000","objects_scrubbed":177,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:55:26.517270+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0.0032490380000000001,"stat_sum":{"num_bytes":14027540,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":18,"num_read_kb":8206,"num_write":307,"num_write_kb":13857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"44'428","reported_seq":2543,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:26.856303+0000","last_change":"2024-03-27T06:22:26.856347+0000","last_active":"2024-03-27T06:22:26.856303+0000","last_peered":"2024-03-27T06:22:26.856303+0000","last_clean":"2024-03-27T06:22:26.856303+0000","last_became_active":"2024-03-27T06:14:03.557224+0000","last_became_peered":"2024-03-27T06:14:03.557224+0000","last_unstale":"2024-03-27T06:22:26.856303+0000","last_undegraded":"2024-03-27T06:22:26.856303+0000","last_fullsized":"2024-03-27T06:22:26.856303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'428","last_scrub_stamp":"2024-03-27T06:22:26.856224+0000","last_deep_scrub":"44'428","last_deep_scrub_stamp":"2024-03-27T06:22:26.856224+0000","last_clean_scrub_stamp":"2024-03-27T06:22:26.856224+0000","objects_scrubbed":153,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:52:14.442246+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0.003068571,"stat_sum":{"num_bytes":18001138,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":16,"num_read_kb":7383,"num_write":307,"num_write_kb":21016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"44'462","reported_seq":2585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.250972+0000","last_change":"2024-03-27T06:22:23.250972+0000","last_active":"2024-03-27T06:22:23.250972+0000","last_peered":"2024-03-27T06:22:23.250972+0000","last_clean":"2024-03-27T06:22:23.250972+0000","last_became_active":"2024-03-27T06:14:03.524418+0000","last_became_peered":"2024-03-27T06:14:03.524418+0000","last_unstale":"2024-03-27T06:22:23.250972+0000","last_undegraded":"2024-03-27T06:22:23.250972+0000","last_fullsized":"2024-03-27T06:22:23.250972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'462","last_scrub_stamp":"2024-03-27T06:22:23.250825+0000","last_deep_scrub":"44'462","last_deep_scrub_stamp":"2024-03-27T06:22:23.250825+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.250825+0000","objects_scrubbed":165,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:00:31.780777+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0.0036751610000000001,"stat_sum":{"num_bytes":22534890,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":22,"num_read_kb":12306,"num_write":351,"num_write_kb":22191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"44'482","reported_seq":2607,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:16.740162+0000","last_change":"2024-03-27T06:22:16.740162+0000","last_active":"2024-03-27T06:22:16.740162+0000","last_peered":"2024-03-27T06:22:16.740162+0000","last_clean":"2024-03-27T06:22:16.740162+0000","last_became_active":"2024-03-27T06:14:03.539178+0000","last_became_peered":"2024-03-27T06:14:03.539178+0000","last_unstale":"2024-03-27T06:22:16.740162+0000","last_undegraded":"2024-03-27T06:22:16.740162+0000","last_fullsized":"2024-03-27T06:22:16.740162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'482","last_scrub_stamp":"2024-03-27T06:22:16.740051+0000","last_deep_scrub":"44'482","last_deep_scrub_stamp":"2024-03-27T06:22:16.740051+0000","last_clean_scrub_stamp":"2024-03-27T06:22:16.740051+0000","objects_scrubbed":163,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:02:39.973708+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0.0075044159999999999,"stat_sum":{"num_bytes":14250100,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":24,"num_read_kb":7593,"num_write":351,"num_write_kb":14104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"44'484","reported_seq":2605,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:22.543216+0000","last_change":"2024-03-27T06:22:22.543216+0000","last_active":"2024-03-27T06:22:22.543216+0000","last_peered":"2024-03-27T06:22:22.543216+0000","last_clean":"2024-03-27T06:22:22.543216+0000","last_became_active":"2024-03-27T06:14:03.527750+0000","last_became_peered":"2024-03-27T06:14:03.527750+0000","last_unstale":"2024-03-27T06:22:22.543216+0000","last_undegraded":"2024-03-27T06:22:22.543216+0000","last_fullsized":"2024-03-27T06:22:22.543216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'484","last_scrub_stamp":"2024-03-27T06:22:22.543084+0000","last_deep_scrub":"44'484","last_deep_scrub_stamp":"2024-03-27T06:22:22.543084+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.543084+0000","objects_scrubbed":177,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:41:56.442910+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0.0011795639999999999,"stat_sum":{"num_bytes":10303903,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":19,"num_read_kb":1482,"num_write":377,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"44'479","reported_seq":2609,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:16.359914+0000","last_change":"2024-03-27T06:22:16.359914+0000","last_active":"2024-03-27T06:22:16.359914+0000","last_peered":"2024-03-27T06:22:16.359914+0000","last_clean":"2024-03-27T06:22:16.359914+0000","last_became_active":"2024-03-27T06:14:03.526403+0000","last_became_peered":"2024-03-27T06:14:03.526403+0000","last_unstale":"2024-03-27T06:22:16.359914+0000","last_undegraded":"2024-03-27T06:22:16.359914+0000","last_fullsized":"2024-03-27T06:22:16.359914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'479","last_scrub_stamp":"2024-03-27T06:22:16.359750+0000","last_deep_scrub":"44'479","last_deep_scrub_stamp":"2024-03-27T06:22:16.359750+0000","last_clean_scrub_stamp":"2024-03-27T06:22:16.359750+0000","objects_scrubbed":178,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:12:42.741325+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.0016911179999999999,"stat_sum":{"num_bytes":11109721,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":28,"num_read_kb":4123,"num_write":370,"num_write_kb":11044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"44'438","reported_seq":2556,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:16.453798+0000","last_change":"2024-03-27T06:22:16.453881+0000","last_active":"2024-03-27T06:22:16.453798+0000","last_peered":"2024-03-27T06:22:16.453798+0000","last_clean":"2024-03-27T06:22:16.453798+0000","last_became_active":"2024-03-27T06:14:03.534380+0000","last_became_peered":"2024-03-27T06:14:03.534380+0000","last_unstale":"2024-03-27T06:22:16.453798+0000","last_undegraded":"2024-03-27T06:22:16.453798+0000","last_fullsized":"2024-03-27T06:22:16.453798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'438","last_scrub_stamp":"2024-03-27T06:22:16.453675+0000","last_deep_scrub":"44'438","last_deep_scrub_stamp":"2024-03-27T06:22:16.453675+0000","last_clean_scrub_stamp":"2024-03-27T06:22:16.453675+0000","objects_scrubbed":164,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:41:50.353501+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0.0057058689999999997,"stat_sum":{"num_bytes":18559363,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":18,"num_read_kb":4112,"num_write":320,"num_write_kb":18284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"44'23","reported_seq":99,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.082170+0000","last_change":"2024-03-27T06:22:19.082220+0000","last_active":"2024-03-27T06:22:19.082170+0000","last_peered":"2024-03-27T06:22:19.082170+0000","last_clean":"2024-03-27T06:22:19.082170+0000","last_became_active":"2024-03-27T06:14:02.522598+0000","last_became_peered":"2024-03-27T06:14:02.522598+0000","last_unstale":"2024-03-27T06:22:19.082170+0000","last_undegraded":"2024-03-27T06:22:19.082170+0000","last_fullsized":"2024-03-27T06:22:19.082170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'23","last_scrub_stamp":"2024-03-27T06:22:19.082083+0000","last_deep_scrub":"44'23","last_deep_scrub_stamp":"2024-03-27T06:22:19.082083+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.082083+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:16:05.229090+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195437,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":19,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20328,"num_omap_keys":34,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'430","reported_seq":2547,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:27.027565+0000","last_change":"2024-03-27T06:22:27.027587+0000","last_active":"2024-03-27T06:22:27.027565+0000","last_peered":"2024-03-27T06:22:27.027565+0000","last_clean":"2024-03-27T06:22:27.027565+0000","last_became_active":"2024-03-27T06:14:03.547586+0000","last_became_peered":"2024-03-27T06:14:03.547586+0000","last_unstale":"2024-03-27T06:22:27.027565+0000","last_undegraded":"2024-03-27T06:22:27.027565+0000","last_fullsized":"2024-03-27T06:22:27.027565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'430","last_scrub_stamp":"2024-03-27T06:22:27.027500+0000","last_deep_scrub":"44'430","last_deep_scrub_stamp":"2024-03-27T06:22:27.027500+0000","last_clean_scrub_stamp":"2024-03-27T06:22:27.027500+0000","objects_scrubbed":150,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:00:39.471209+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0.0018454000000000001,"stat_sum":{"num_bytes":6743271,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":310,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"44'258","reported_seq":340,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:27.594535+0000","last_change":"2024-03-27T06:22:27.594566+0000","last_active":"2024-03-27T06:22:27.594535+0000","last_peered":"2024-03-27T06:22:27.594535+0000","last_clean":"2024-03-27T06:22:27.594535+0000","last_became_active":"2024-03-27T06:14:02.540191+0000","last_became_peered":"2024-03-27T06:14:02.540191+0000","last_unstale":"2024-03-27T06:22:27.594535+0000","last_undegraded":"2024-03-27T06:22:27.594535+0000","last_fullsized":"2024-03-27T06:22:27.594535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'258","last_scrub_stamp":"2024-03-27T06:22:27.594466+0000","last_deep_scrub":"44'258","last_deep_scrub_stamp":"2024-03-27T06:22:27.594466+0000","last_clean_scrub_stamp":"2024-03-27T06:22:27.594466+0000","objects_scrubbed":5,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:15:00.463781+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":8577,"num_write":252,"num_write_kb":12472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7787,"num_omap_keys":13,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"44'67","reported_seq":146,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.740145+0000","last_change":"2024-03-27T06:22:19.740173+0000","last_active":"2024-03-27T06:22:19.740145+0000","last_peered":"2024-03-27T06:22:19.740145+0000","last_clean":"2024-03-27T06:22:19.740145+0000","last_became_active":"2024-03-27T06:14:02.534534+0000","last_became_peered":"2024-03-27T06:14:02.534534+0000","last_unstale":"2024-03-27T06:22:19.740145+0000","last_undegraded":"2024-03-27T06:22:19.740145+0000","last_fullsized":"2024-03-27T06:22:19.740145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'67","last_scrub_stamp":"2024-03-27T06:22:19.740084+0000","last_deep_scrub":"44'67","last_deep_scrub_stamp":"2024-03-27T06:22:19.740084+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.740084+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:23:18.824123+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":4114,"num_write":65,"num_write_kb":4173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27617,"num_omap_keys":47,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"44'517","reported_seq":2645,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.292086+0000","last_change":"2024-03-27T06:22:24.292142+0000","last_active":"2024-03-27T06:22:24.292086+0000","last_peered":"2024-03-27T06:22:24.292086+0000","last_clean":"2024-03-27T06:22:24.292086+0000","last_became_active":"2024-03-27T06:14:03.525034+0000","last_became_peered":"2024-03-27T06:14:03.525034+0000","last_unstale":"2024-03-27T06:22:24.292086+0000","last_undegraded":"2024-03-27T06:22:24.292086+0000","last_fullsized":"2024-03-27T06:22:24.292086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'517","last_scrub_stamp":"2024-03-27T06:22:24.291977+0000","last_deep_scrub":"44'517","last_deep_scrub_stamp":"2024-03-27T06:22:24.291977+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.291977+0000","objects_scrubbed":188,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:49:27.244633+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0.003103719,"stat_sum":{"num_bytes":18931276,"num_objects":188,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":28,"num_read_kb":12930,"num_write":396,"num_write_kb":26887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"44'381","reported_seq":455,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.601097+0000","last_change":"2024-03-27T06:22:24.601157+0000","last_active":"2024-03-27T06:22:24.601097+0000","last_peered":"2024-03-27T06:22:24.601097+0000","last_clean":"2024-03-27T06:22:24.601097+0000","last_became_active":"2024-03-27T06:14:02.535032+0000","last_became_peered":"2024-03-27T06:14:02.535032+0000","last_unstale":"2024-03-27T06:22:24.601097+0000","last_undegraded":"2024-03-27T06:22:24.601097+0000","last_fullsized":"2024-03-27T06:22:24.601097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'381","last_scrub_stamp":"2024-03-27T06:22:24.601007+0000","last_deep_scrub":"44'381","last_deep_scrub_stamp":"2024-03-27T06:22:24.601007+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.601007+0000","objects_scrubbed":5,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:26:35.314663+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":374,"num_write_kb":12507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25120,"num_omap_keys":42,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"44'458","reported_seq":2580,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:25.076409+0000","last_change":"2024-03-27T06:22:25.076471+0000","last_active":"2024-03-27T06:22:25.076409+0000","last_peered":"2024-03-27T06:22:25.076409+0000","last_clean":"2024-03-27T06:22:25.076409+0000","last_became_active":"2024-03-27T06:14:03.530626+0000","last_became_peered":"2024-03-27T06:14:03.530626+0000","last_unstale":"2024-03-27T06:22:25.076409+0000","last_undegraded":"2024-03-27T06:22:25.076409+0000","last_fullsized":"2024-03-27T06:22:25.076409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'458","last_scrub_stamp":"2024-03-27T06:22:25.076322+0000","last_deep_scrub":"44'458","last_deep_scrub_stamp":"2024-03-27T06:22:25.076322+0000","last_clean_scrub_stamp":"2024-03-27T06:22:25.076322+0000","objects_scrubbed":171,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:14:05.750144+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0019163050000000001,"stat_sum":{"num_bytes":19664158,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":13845,"num_write":329,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"44'14","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.299712+0000","last_change":"2024-03-27T06:22:15.299761+0000","last_active":"2024-03-27T06:22:15.299712+0000","last_peered":"2024-03-27T06:22:15.299712+0000","last_clean":"2024-03-27T06:22:15.299712+0000","last_became_active":"2024-03-27T06:14:02.524656+0000","last_became_peered":"2024-03-27T06:14:02.524656+0000","last_unstale":"2024-03-27T06:22:15.299712+0000","last_undegraded":"2024-03-27T06:22:15.299712+0000","last_fullsized":"2024-03-27T06:22:15.299712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'14","last_scrub_stamp":"2024-03-27T06:22:15.299611+0000","last_deep_scrub":"44'14","last_deep_scrub_stamp":"2024-03-27T06:22:15.299611+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.299611+0000","objects_scrubbed":4,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:20:07.902308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5952,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"44'460","reported_seq":2585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.304737+0000","last_change":"2024-03-27T06:22:20.304797+0000","last_active":"2024-03-27T06:22:20.304737+0000","last_peered":"2024-03-27T06:22:20.304737+0000","last_clean":"2024-03-27T06:22:20.304737+0000","last_became_active":"2024-03-27T06:14:03.543477+0000","last_became_peered":"2024-03-27T06:14:03.543477+0000","last_unstale":"2024-03-27T06:22:20.304737+0000","last_undegraded":"2024-03-27T06:22:20.304737+0000","last_fullsized":"2024-03-27T06:22:20.304737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'460","last_scrub_stamp":"2024-03-27T06:22:20.304657+0000","last_deep_scrub":"44'460","last_deep_scrub_stamp":"2024-03-27T06:22:20.304657+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.304657+0000","objects_scrubbed":163,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:18:14.821330+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0.0026075790000000001,"stat_sum":{"num_bytes":22946854,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":25,"num_read_kb":12308,"num_write":351,"num_write_kb":22593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"44'111","reported_seq":186,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.074537+0000","last_change":"2024-03-27T06:22:14.074616+0000","last_active":"2024-03-27T06:22:14.074537+0000","last_peered":"2024-03-27T06:22:14.074537+0000","last_clean":"2024-03-27T06:22:14.074537+0000","last_became_active":"2024-03-27T06:14:02.528964+0000","last_became_peered":"2024-03-27T06:14:02.528964+0000","last_unstale":"2024-03-27T06:22:14.074537+0000","last_undegraded":"2024-03-27T06:22:14.074537+0000","last_fullsized":"2024-03-27T06:22:14.074537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'111","last_scrub_stamp":"2024-03-27T06:22:14.074335+0000","last_deep_scrub":"44'111","last_deep_scrub_stamp":"2024-03-27T06:22:14.074335+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.074335+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:10:24.848318+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":106,"num_write_kb":4193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24559,"num_omap_keys":41,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"44'445","reported_seq":2567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:31.246943+0000","last_change":"2024-03-27T06:22:31.246997+0000","last_active":"2024-03-27T06:22:31.246943+0000","last_peered":"2024-03-27T06:22:31.246943+0000","last_clean":"2024-03-27T06:22:31.246943+0000","last_became_active":"2024-03-27T06:14:03.542102+0000","last_became_peered":"2024-03-27T06:14:03.542102+0000","last_unstale":"2024-03-27T06:22:31.246943+0000","last_undegraded":"2024-03-27T06:22:31.246943+0000","last_fullsized":"2024-03-27T06:22:31.246943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'445","last_scrub_stamp":"2024-03-27T06:22:31.246838+0000","last_deep_scrub":"44'445","last_deep_scrub_stamp":"2024-03-27T06:22:31.246838+0000","last_clean_scrub_stamp":"2024-03-27T06:22:31.246838+0000","objects_scrubbed":157,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:15:04.116153+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0.003650304,"stat_sum":{"num_bytes":10082281,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":22,"num_read_kb":4115,"num_write":319,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"44'287","reported_seq":362,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.359454+0000","last_change":"2024-03-27T06:22:18.359522+0000","last_active":"2024-03-27T06:22:18.359454+0000","last_peered":"2024-03-27T06:22:18.359454+0000","last_clean":"2024-03-27T06:22:18.359454+0000","last_became_active":"2024-03-27T06:14:02.526661+0000","last_became_peered":"2024-03-27T06:14:02.526661+0000","last_unstale":"2024-03-27T06:22:18.359454+0000","last_undegraded":"2024-03-27T06:22:18.359454+0000","last_fullsized":"2024-03-27T06:22:18.359454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'287","last_scrub_stamp":"2024-03-27T06:22:18.359385+0000","last_deep_scrub":"44'287","last_deep_scrub_stamp":"2024-03-27T06:22:18.359385+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.359385+0000","objects_scrubbed":7,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:52:52.109824+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":284,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":39372,"num_omap_keys":68,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'486","reported_seq":2602,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:32.815968+0000","last_change":"2024-03-27T06:22:32.816006+0000","last_active":"2024-03-27T06:22:32.815968+0000","last_peered":"2024-03-27T06:22:32.815968+0000","last_clean":"2024-03-27T06:22:32.815968+0000","last_became_active":"2024-03-27T06:14:03.546424+0000","last_became_peered":"2024-03-27T06:14:03.546424+0000","last_unstale":"2024-03-27T06:22:32.815968+0000","last_undegraded":"2024-03-27T06:22:32.815968+0000","last_fullsized":"2024-03-27T06:22:32.815968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'486","last_scrub_stamp":"2024-03-27T06:22:32.815898+0000","last_deep_scrub":"44'486","last_deep_scrub_stamp":"2024-03-27T06:22:32.815898+0000","last_clean_scrub_stamp":"2024-03-27T06:22:32.815898+0000","objects_scrubbed":162,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:44:48.396405+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0.0084315699999999993,"stat_sum":{"num_bytes":18808790,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":315,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"44'134","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.280944+0000","last_change":"2024-03-27T06:22:14.280944+0000","last_active":"2024-03-27T06:22:14.280944+0000","last_peered":"2024-03-27T06:22:14.280944+0000","last_clean":"2024-03-27T06:22:14.280944+0000","last_became_active":"2024-03-27T06:14:02.516057+0000","last_became_peered":"2024-03-27T06:14:02.516057+0000","last_unstale":"2024-03-27T06:22:14.280944+0000","last_undegraded":"2024-03-27T06:22:14.280944+0000","last_fullsized":"2024-03-27T06:22:14.280944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'133","last_scrub_stamp":"2024-03-27T06:22:14.280778+0000","last_deep_scrub":"44'133","last_deep_scrub_stamp":"2024-03-27T06:22:14.280778+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.280778+0000","objects_scrubbed":7,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:36:18.758700+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":8203,"num_write":124,"num_write_kb":12365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"44'442","reported_seq":2561,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.245050+0000","last_change":"2024-03-27T06:22:24.245077+0000","last_active":"2024-03-27T06:22:24.245050+0000","last_peered":"2024-03-27T06:22:24.245050+0000","last_clean":"2024-03-27T06:22:24.245050+0000","last_became_active":"2024-03-27T06:14:03.551523+0000","last_became_peered":"2024-03-27T06:14:03.551523+0000","last_unstale":"2024-03-27T06:22:24.245050+0000","last_undegraded":"2024-03-27T06:22:24.245050+0000","last_fullsized":"2024-03-27T06:22:24.245050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'442","last_scrub_stamp":"2024-03-27T06:22:24.244958+0000","last_deep_scrub":"44'442","last_deep_scrub_stamp":"2024-03-27T06:22:24.244958+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.244958+0000","objects_scrubbed":175,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:48:38.487684+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0.0050386370000000003,"stat_sum":{"num_bytes":2468399,"num_objects":175,"num_object_clones":0,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":327,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"44'151","reported_seq":235,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.381975+0000","last_change":"2024-03-27T06:22:20.382015+0000","last_active":"2024-03-27T06:22:20.381975+0000","last_peered":"2024-03-27T06:22:20.381975+0000","last_clean":"2024-03-27T06:22:20.381975+0000","last_became_active":"2024-03-27T06:14:02.533492+0000","last_became_peered":"2024-03-27T06:14:02.533492+0000","last_unstale":"2024-03-27T06:22:20.381975+0000","last_undegraded":"2024-03-27T06:22:20.381975+0000","last_fullsized":"2024-03-27T06:22:20.381975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'151","last_scrub_stamp":"2024-03-27T06:22:20.381912+0000","last_deep_scrub":"44'151","last_deep_scrub_stamp":"2024-03-27T06:22:20.381912+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.381912+0000","objects_scrubbed":8,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:41:23.182763+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":8224,"num_write":147,"num_write_kb":8324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":39142,"num_omap_keys":70,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'468","reported_seq":2593,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.840499+0000","last_change":"2024-03-27T06:22:24.840549+0000","last_active":"2024-03-27T06:22:24.840499+0000","last_peered":"2024-03-27T06:22:24.840499+0000","last_clean":"2024-03-27T06:22:24.840499+0000","last_became_active":"2024-03-27T06:14:03.537959+0000","last_became_peered":"2024-03-27T06:14:03.537959+0000","last_unstale":"2024-03-27T06:22:24.840499+0000","last_undegraded":"2024-03-27T06:22:24.840499+0000","last_fullsized":"2024-03-27T06:22:24.840499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'468","last_scrub_stamp":"2024-03-27T06:22:24.840404+0000","last_deep_scrub":"44'468","last_deep_scrub_stamp":"2024-03-27T06:22:24.840404+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.840404+0000","objects_scrubbed":184,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:08:09.433847+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0.002905866,"stat_sum":{"num_bytes":19666163,"num_objects":184,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":25,"num_read_kb":16403,"num_write":336,"num_write_kb":27577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"44'8","reported_seq":82,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:26.535137+0000","last_change":"2024-03-27T06:22:26.535196+0000","last_active":"2024-03-27T06:22:26.535137+0000","last_peered":"2024-03-27T06:22:26.535137+0000","last_clean":"2024-03-27T06:22:26.535137+0000","last_became_active":"2024-03-27T06:14:02.537390+0000","last_became_peered":"2024-03-27T06:14:02.537390+0000","last_unstale":"2024-03-27T06:22:26.535137+0000","last_undegraded":"2024-03-27T06:22:26.535137+0000","last_fullsized":"2024-03-27T06:22:26.535137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'8","last_scrub_stamp":"2024-03-27T06:22:26.535013+0000","last_deep_scrub":"44'8","last_deep_scrub_stamp":"2024-03-27T06:22:26.535013+0000","last_clean_scrub_stamp":"2024-03-27T06:22:26.535013+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:09:11.719060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":38797,"num_omap_keys":67,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"44'532","reported_seq":2655,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:34.679531+0000","last_change":"2024-03-27T06:22:34.679584+0000","last_active":"2024-03-27T06:22:34.679531+0000","last_peered":"2024-03-27T06:22:34.679531+0000","last_clean":"2024-03-27T06:22:34.679531+0000","last_became_active":"2024-03-27T06:14:03.555681+0000","last_became_peered":"2024-03-27T06:14:03.555681+0000","last_unstale":"2024-03-27T06:22:34.679531+0000","last_undegraded":"2024-03-27T06:22:34.679531+0000","last_fullsized":"2024-03-27T06:22:34.679531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'532","last_scrub_stamp":"2024-03-27T06:22:34.679446+0000","last_deep_scrub":"44'532","last_deep_scrub_stamp":"2024-03-27T06:22:34.679446+0000","last_clean_scrub_stamp":"2024-03-27T06:22:34.679446+0000","objects_scrubbed":171,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:57:23.734205+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0.010165321999999999,"stat_sum":{"num_bytes":10617237,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":345,"num_write_kb":10547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"44'16","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.006948+0000","last_change":"2024-03-27T06:22:18.007008+0000","last_active":"2024-03-27T06:22:18.006948+0000","last_peered":"2024-03-27T06:22:18.006948+0000","last_clean":"2024-03-27T06:22:18.006948+0000","last_became_active":"2024-03-27T06:14:02.541164+0000","last_became_peered":"2024-03-27T06:14:02.541164+0000","last_unstale":"2024-03-27T06:22:18.006948+0000","last_undegraded":"2024-03-27T06:22:18.006948+0000","last_fullsized":"2024-03-27T06:22:18.006948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'16","last_scrub_stamp":"2024-03-27T06:22:18.006843+0000","last_deep_scrub":"44'16","last_deep_scrub_stamp":"2024-03-27T06:22:18.006843+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.006843+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:42:53.084592+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1215,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":16,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":31900,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"44'468","reported_seq":2584,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.467309+0000","last_change":"2024-03-27T06:22:17.467385+0000","last_active":"2024-03-27T06:22:17.467309+0000","last_peered":"2024-03-27T06:22:17.467309+0000","last_clean":"2024-03-27T06:22:17.467309+0000","last_became_active":"2024-03-27T06:14:03.536054+0000","last_became_peered":"2024-03-27T06:14:03.536054+0000","last_unstale":"2024-03-27T06:22:17.467309+0000","last_undegraded":"2024-03-27T06:22:17.467309+0000","last_fullsized":"2024-03-27T06:22:17.467309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'468","last_scrub_stamp":"2024-03-27T06:22:17.467229+0000","last_deep_scrub":"44'468","last_deep_scrub_stamp":"2024-03-27T06:22:17.467229+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.467229+0000","objects_scrubbed":174,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:34:24.714988+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0.0054212890000000001,"stat_sum":{"num_bytes":24812922,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":17,"num_read_kb":8206,"num_write":342,"num_write_kb":26402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"44'294","reported_seq":384,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:16.870178+0000","last_change":"2024-03-27T06:22:16.870220+0000","last_active":"2024-03-27T06:22:16.870178+0000","last_peered":"2024-03-27T06:22:16.870178+0000","last_clean":"2024-03-27T06:22:16.870178+0000","last_became_active":"2024-03-27T06:14:02.522875+0000","last_became_peered":"2024-03-27T06:14:02.522875+0000","last_unstale":"2024-03-27T06:22:16.870178+0000","last_undegraded":"2024-03-27T06:22:16.870178+0000","last_fullsized":"2024-03-27T06:22:16.870178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'294","last_scrub_stamp":"2024-03-27T06:22:16.870105+0000","last_deep_scrub":"44'294","last_deep_scrub_stamp":"2024-03-27T06:22:16.870105+0000","last_clean_scrub_stamp":"2024-03-27T06:22:16.870105+0000","objects_scrubbed":17,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:39:43.105757+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":283,"num_write_kb":8391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66188,"num_omap_keys":112,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"44'505","reported_seq":2623,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:28.764155+0000","last_change":"2024-03-27T06:22:28.764196+0000","last_active":"2024-03-27T06:22:28.764155+0000","last_peered":"2024-03-27T06:22:28.764155+0000","last_clean":"2024-03-27T06:22:28.764155+0000","last_became_active":"2024-03-27T06:14:03.539535+0000","last_became_peered":"2024-03-27T06:14:03.539535+0000","last_unstale":"2024-03-27T06:22:28.764155+0000","last_undegraded":"2024-03-27T06:22:28.764155+0000","last_fullsized":"2024-03-27T06:22:28.764155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'505","last_scrub_stamp":"2024-03-27T06:22:28.764050+0000","last_deep_scrub":"44'505","last_deep_scrub_stamp":"2024-03-27T06:22:28.764050+0000","last_clean_scrub_stamp":"2024-03-27T06:22:28.764050+0000","objects_scrubbed":178,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:04:46.588095+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.0083622499999999999,"stat_sum":{"num_bytes":12063538,"num_objects":178,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":20,"num_read_kb":12305,"num_write":353,"num_write_kb":20111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"44'32","reported_seq":109,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.030143+0000","last_change":"2024-03-27T06:22:17.030201+0000","last_active":"2024-03-27T06:22:17.030143+0000","last_peered":"2024-03-27T06:22:17.030143+0000","last_clean":"2024-03-27T06:22:17.030143+0000","last_became_active":"2024-03-27T06:14:02.537183+0000","last_became_peered":"2024-03-27T06:14:02.537183+0000","last_unstale":"2024-03-27T06:22:17.030143+0000","last_undegraded":"2024-03-27T06:22:17.030143+0000","last_fullsized":"2024-03-27T06:22:17.030143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'32","last_scrub_stamp":"2024-03-27T06:22:17.030039+0000","last_deep_scrub":"44'32","last_deep_scrub_stamp":"2024-03-27T06:22:17.030039+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.030039+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:22:34.168748+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4103,"num_write":34,"num_write_kb":4119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9584,"num_omap_keys":16,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'391","reported_seq":2504,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.285798+0000","last_change":"2024-03-27T06:22:21.285907+0000","last_active":"2024-03-27T06:22:21.285798+0000","last_peered":"2024-03-27T06:22:21.285798+0000","last_clean":"2024-03-27T06:22:21.285798+0000","last_became_active":"2024-03-27T06:14:03.552421+0000","last_became_peered":"2024-03-27T06:14:03.552421+0000","last_unstale":"2024-03-27T06:22:21.285798+0000","last_undegraded":"2024-03-27T06:22:21.285798+0000","last_fullsized":"2024-03-27T06:22:21.285798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'391","last_scrub_stamp":"2024-03-27T06:22:21.285673+0000","last_deep_scrub":"44'391","last_deep_scrub_stamp":"2024-03-27T06:22:21.285673+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.285673+0000","objects_scrubbed":166,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:11:28.668910+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0.0022904700000000002,"stat_sum":{"num_bytes":11066744,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":12134,"num_write":288,"num_write_kb":15054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"44'31","reported_seq":105,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.882677+0000","last_change":"2024-03-27T06:22:15.882706+0000","last_active":"2024-03-27T06:22:15.882677+0000","last_peered":"2024-03-27T06:22:15.882677+0000","last_clean":"2024-03-27T06:22:15.882677+0000","last_became_active":"2024-03-27T06:14:02.516458+0000","last_became_peered":"2024-03-27T06:14:02.516458+0000","last_unstale":"2024-03-27T06:22:15.882677+0000","last_undegraded":"2024-03-27T06:22:15.882677+0000","last_fullsized":"2024-03-27T06:22:15.882677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'31","last_scrub_stamp":"2024-03-27T06:22:15.882609+0000","last_deep_scrub":"44'31","last_deep_scrub_stamp":"2024-03-27T06:22:15.882609+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.882609+0000","objects_scrubbed":4,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:38:31.490474+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":30,"num_write_kb":4134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26367,"num_omap_keys":45,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"44'435","reported_seq":2558,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:25.184164+0000","last_change":"2024-03-27T06:22:25.184206+0000","last_active":"2024-03-27T06:22:25.184164+0000","last_peered":"2024-03-27T06:22:25.184164+0000","last_clean":"2024-03-27T06:22:25.184164+0000","last_became_active":"2024-03-27T06:14:03.548242+0000","last_became_peered":"2024-03-27T06:14:03.548242+0000","last_unstale":"2024-03-27T06:22:25.184164+0000","last_undegraded":"2024-03-27T06:22:25.184164+0000","last_fullsized":"2024-03-27T06:22:25.184164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'435","last_scrub_stamp":"2024-03-27T06:22:25.184098+0000","last_deep_scrub":"44'435","last_deep_scrub_stamp":"2024-03-27T06:22:25.184098+0000","last_clean_scrub_stamp":"2024-03-27T06:22:25.184098+0000","objects_scrubbed":168,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:24:40.652658+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0.0050322370000000002,"stat_sum":{"num_bytes":14587209,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":24,"num_read_kb":12307,"num_write":322,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"44'25","reported_seq":99,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.136052+0000","last_change":"2024-03-27T06:22:20.136096+0000","last_active":"2024-03-27T06:22:20.136052+0000","last_peered":"2024-03-27T06:22:20.136052+0000","last_clean":"2024-03-27T06:22:20.136052+0000","last_became_active":"2024-03-27T06:14:02.528479+0000","last_became_peered":"2024-03-27T06:14:02.528479+0000","last_unstale":"2024-03-27T06:22:20.136052+0000","last_undegraded":"2024-03-27T06:22:20.136052+0000","last_fullsized":"2024-03-27T06:22:20.136052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'25","last_scrub_stamp":"2024-03-27T06:22:20.135940+0000","last_deep_scrub":"44'25","last_deep_scrub_stamp":"2024-03-27T06:22:20.135940+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.135940+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:54:06.327072+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":23,"num_write_kb":4120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13679,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"44'469","reported_seq":2586,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.251339+0000","last_change":"2024-03-27T06:22:21.251366+0000","last_active":"2024-03-27T06:22:21.251339+0000","last_peered":"2024-03-27T06:22:21.251339+0000","last_clean":"2024-03-27T06:22:21.251339+0000","last_became_active":"2024-03-27T06:14:03.548127+0000","last_became_peered":"2024-03-27T06:14:03.548127+0000","last_unstale":"2024-03-27T06:22:21.251339+0000","last_undegraded":"2024-03-27T06:22:21.251339+0000","last_fullsized":"2024-03-27T06:22:21.251339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'469","last_scrub_stamp":"2024-03-27T06:22:21.251249+0000","last_deep_scrub":"44'469","last_deep_scrub_stamp":"2024-03-27T06:22:21.251249+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.251249+0000","objects_scrubbed":176,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:14:57.515948+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0.0070052669999999999,"stat_sum":{"num_bytes":11205183,"num_objects":176,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":352,"num_write_kb":11129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"44'33","reported_seq":114,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.780011+0000","last_change":"2024-03-27T06:22:18.780060+0000","last_active":"2024-03-27T06:22:18.780011+0000","last_peered":"2024-03-27T06:22:18.780011+0000","last_clean":"2024-03-27T06:22:18.780011+0000","last_became_active":"2024-03-27T06:14:02.528934+0000","last_became_peered":"2024-03-27T06:14:02.528934+0000","last_unstale":"2024-03-27T06:22:18.780011+0000","last_undegraded":"2024-03-27T06:22:18.780011+0000","last_fullsized":"2024-03-27T06:22:18.780011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'33","last_scrub_stamp":"2024-03-27T06:22:18.779910+0000","last_deep_scrub":"44'33","last_deep_scrub_stamp":"2024-03-27T06:22:18.779910+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.779910+0000","objects_scrubbed":5,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:38:34.387775+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":32,"num_write_kb":4138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20328,"num_omap_keys":34,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"44'446","reported_seq":2566,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:30.753364+0000","last_change":"2024-03-27T06:22:30.753391+0000","last_active":"2024-03-27T06:22:30.753364+0000","last_peered":"2024-03-27T06:22:30.753364+0000","last_clean":"2024-03-27T06:22:30.753364+0000","last_became_active":"2024-03-27T06:14:03.543639+0000","last_became_peered":"2024-03-27T06:14:03.543639+0000","last_unstale":"2024-03-27T06:22:30.753364+0000","last_undegraded":"2024-03-27T06:22:30.753364+0000","last_fullsized":"2024-03-27T06:22:30.753364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'446","last_scrub_stamp":"2024-03-27T06:22:30.753296+0000","last_deep_scrub":"44'446","last_deep_scrub_stamp":"2024-03-27T06:22:30.753296+0000","last_clean_scrub_stamp":"2024-03-27T06:22:30.753296+0000","objects_scrubbed":150,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:36:01.620087+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0.0086058899999999997,"stat_sum":{"num_bytes":14608246,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":19,"num_read_kb":4113,"num_write":322,"num_write_kb":18530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'119","reported_seq":230,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:14.301075+0000","last_change":"2024-03-27T06:22:14.301165+0000","last_active":"2024-03-27T06:22:14.301075+0000","last_peered":"2024-03-27T06:22:14.301075+0000","last_clean":"2024-03-27T06:22:14.301075+0000","last_became_active":"2024-03-27T06:13:58.725636+0000","last_became_peered":"2024-03-27T06:13:58.725636+0000","last_unstale":"2024-03-27T06:22:14.301075+0000","last_undegraded":"2024-03-27T06:22:14.301075+0000","last_fullsized":"2024-03-27T06:22:14.301075+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'119","last_scrub_stamp":"2024-03-27T06:22:14.300945+0000","last_deep_scrub":"44'119","last_deep_scrub_stamp":"2024-03-27T06:22:14.300945+0000","last_clean_scrub_stamp":"2024-03-27T06:22:14.300945+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:41:17.101796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'90","reported_seq":168,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:25.591151+0000","last_change":"2024-03-27T06:22:25.591151+0000","last_active":"2024-03-27T06:22:25.591151+0000","last_peered":"2024-03-27T06:22:25.591151+0000","last_clean":"2024-03-27T06:22:25.591151+0000","last_became_active":"2024-03-27T06:14:02.530547+0000","last_became_peered":"2024-03-27T06:14:02.530547+0000","last_unstale":"2024-03-27T06:22:25.591151+0000","last_undegraded":"2024-03-27T06:22:25.591151+0000","last_fullsized":"2024-03-27T06:22:25.591151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'90","last_scrub_stamp":"2024-03-27T06:22:25.591005+0000","last_deep_scrub":"44'90","last_deep_scrub_stamp":"2024-03-27T06:22:25.591005+0000","last_clean_scrub_stamp":"2024-03-27T06:22:25.591005+0000","objects_scrubbed":7,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T10:08:47.016903+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":12,"num_write":86,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49368,"num_omap_keys":86,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'428","reported_seq":2543,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:26.038916+0000","last_change":"2024-03-27T06:22:26.038970+0000","last_active":"2024-03-27T06:22:26.038916+0000","last_peered":"2024-03-27T06:22:26.038916+0000","last_clean":"2024-03-27T06:22:26.038916+0000","last_became_active":"2024-03-27T06:14:03.541129+0000","last_became_peered":"2024-03-27T06:14:03.541129+0000","last_unstale":"2024-03-27T06:22:26.038916+0000","last_undegraded":"2024-03-27T06:22:26.038916+0000","last_fullsized":"2024-03-27T06:22:26.038916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'428","last_scrub_stamp":"2024-03-27T06:22:26.038812+0000","last_deep_scrub":"44'428","last_deep_scrub_stamp":"2024-03-27T06:22:26.038812+0000","last_clean_scrub_stamp":"2024-03-27T06:22:26.038812+0000","objects_scrubbed":159,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:23:35.921140+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0.002005691,"stat_sum":{"num_bytes":10118185,"num_objects":159,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":14,"num_read_kb":4109,"num_write":319,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"44'235","reported_seq":316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:20.511687+0000","last_change":"2024-03-27T06:22:20.511894+0000","last_active":"2024-03-27T06:22:20.511687+0000","last_peered":"2024-03-27T06:22:20.511687+0000","last_clean":"2024-03-27T06:22:20.511687+0000","last_became_active":"2024-03-27T06:14:02.526903+0000","last_became_peered":"2024-03-27T06:14:02.526903+0000","last_unstale":"2024-03-27T06:22:20.511687+0000","last_undegraded":"2024-03-27T06:22:20.511687+0000","last_fullsized":"2024-03-27T06:22:20.511687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'235","last_scrub_stamp":"2024-03-27T06:22:20.511456+0000","last_deep_scrub":"44'235","last_deep_scrub_stamp":"2024-03-27T06:22:20.511456+0000","last_clean_scrub_stamp":"2024-03-27T06:22:20.511456+0000","objects_scrubbed":7,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:33:36.805738+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":8210,"num_write":227,"num_write_kb":12456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25736,"num_omap_keys":44,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"44'440","reported_seq":2551,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.828975+0000","last_change":"2024-03-27T06:22:23.828975+0000","last_active":"2024-03-27T06:22:23.828975+0000","last_peered":"2024-03-27T06:22:23.828975+0000","last_clean":"2024-03-27T06:22:23.828975+0000","last_became_active":"2024-03-27T06:14:03.526274+0000","last_became_peered":"2024-03-27T06:14:03.526274+0000","last_unstale":"2024-03-27T06:22:23.828975+0000","last_undegraded":"2024-03-27T06:22:23.828975+0000","last_fullsized":"2024-03-27T06:22:23.828975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'440","last_scrub_stamp":"2024-03-27T06:22:23.828828+0000","last_deep_scrub":"44'440","last_deep_scrub_stamp":"2024-03-27T06:22:23.828828+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.828828+0000","objects_scrubbed":174,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:44:08.425886+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0.001779484,"stat_sum":{"num_bytes":10043455,"num_objects":174,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":324,"num_write_kb":9981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"44'95","reported_seq":256,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.366520+0000","last_change":"2024-03-27T06:22:19.366520+0000","last_active":"2024-03-27T06:22:19.366520+0000","last_peered":"2024-03-27T06:22:19.366520+0000","last_clean":"2024-03-27T06:22:19.366520+0000","last_became_active":"2024-03-27T06:14:02.526724+0000","last_became_peered":"2024-03-27T06:14:02.526724+0000","last_unstale":"2024-03-27T06:22:19.366520+0000","last_undegraded":"2024-03-27T06:22:19.366520+0000","last_fullsized":"2024-03-27T06:22:19.366520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'95","last_scrub_stamp":"2024-03-27T06:22:19.366373+0000","last_deep_scrub":"44'95","last_deep_scrub_stamp":"2024-03-27T06:22:19.366373+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.366373+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:19:37.421219+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534287,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":595,"num_write":90,"num_write_kb":602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21564,"num_omap_keys":36,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"44'451","reported_seq":2570,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.236699+0000","last_change":"2024-03-27T06:22:21.236753+0000","last_active":"2024-03-27T06:22:21.236699+0000","last_peered":"2024-03-27T06:22:21.236699+0000","last_clean":"2024-03-27T06:22:21.236699+0000","last_became_active":"2024-03-27T06:14:03.522418+0000","last_became_peered":"2024-03-27T06:14:03.522418+0000","last_unstale":"2024-03-27T06:22:21.236699+0000","last_undegraded":"2024-03-27T06:22:21.236699+0000","last_fullsized":"2024-03-27T06:22:21.236699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'451","last_scrub_stamp":"2024-03-27T06:22:21.236617+0000","last_deep_scrub":"44'451","last_deep_scrub_stamp":"2024-03-27T06:22:21.236617+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.236617+0000","objects_scrubbed":182,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:13:11.045727+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0.0031179929999999999,"stat_sum":{"num_bytes":2779321,"num_objects":182,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":332,"num_write_kb":2894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"44'43","reported_seq":118,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.508204+0000","last_change":"2024-03-27T06:22:21.508204+0000","last_active":"2024-03-27T06:22:21.508204+0000","last_peered":"2024-03-27T06:22:21.508204+0000","last_clean":"2024-03-27T06:22:21.508204+0000","last_became_active":"2024-03-27T06:14:02.529095+0000","last_became_peered":"2024-03-27T06:14:02.529095+0000","last_unstale":"2024-03-27T06:22:21.508204+0000","last_undegraded":"2024-03-27T06:22:21.508204+0000","last_fullsized":"2024-03-27T06:22:21.508204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'43","last_scrub_stamp":"2024-03-27T06:22:21.508017+0000","last_deep_scrub":"44'43","last_deep_scrub_stamp":"2024-03-27T06:22:21.508017+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.508017+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:59:42.768465+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":36,"num_write_kb":8227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18238,"num_omap_keys":32,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"44'447","reported_seq":2556,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.593510+0000","last_change":"2024-03-27T06:22:24.593560+0000","last_active":"2024-03-27T06:22:24.593510+0000","last_peered":"2024-03-27T06:22:24.593510+0000","last_clean":"2024-03-27T06:22:24.593510+0000","last_became_active":"2024-03-27T06:14:49.853879+0000","last_became_peered":"2024-03-27T06:14:49.853879+0000","last_unstale":"2024-03-27T06:22:24.593510+0000","last_undegraded":"2024-03-27T06:22:24.593510+0000","last_fullsized":"2024-03-27T06:22:24.593510+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'447","last_scrub_stamp":"2024-03-27T06:22:24.593411+0000","last_deep_scrub":"44'447","last_deep_scrub_stamp":"2024-03-27T06:22:24.593411+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.593411+0000","objects_scrubbed":164,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:27:03.888428+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0.0022092909999999999,"stat_sum":{"num_bytes":5308376,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":346,"num_write_kb":5365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"44'81","reported_seq":154,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.140863+0000","last_change":"2024-03-27T06:22:17.140863+0000","last_active":"2024-03-27T06:22:17.140863+0000","last_peered":"2024-03-27T06:22:17.140863+0000","last_clean":"2024-03-27T06:22:17.140863+0000","last_became_active":"2024-03-27T06:14:02.518571+0000","last_became_peered":"2024-03-27T06:14:02.518571+0000","last_unstale":"2024-03-27T06:22:17.140863+0000","last_undegraded":"2024-03-27T06:22:17.140863+0000","last_fullsized":"2024-03-27T06:22:17.140863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'81","last_scrub_stamp":"2024-03-27T06:22:17.140688+0000","last_deep_scrub":"44'81","last_deep_scrub_stamp":"2024-03-27T06:22:17.140688+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.140688+0000","objects_scrubbed":3,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:50:18.018197+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":4150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19767,"num_omap_keys":33,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"44'465","reported_seq":2590,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.428493+0000","last_change":"2024-03-27T06:22:15.428493+0000","last_active":"2024-03-27T06:22:15.428493+0000","last_peered":"2024-03-27T06:22:15.428493+0000","last_clean":"2024-03-27T06:22:15.428493+0000","last_became_active":"2024-03-27T06:14:03.525657+0000","last_became_peered":"2024-03-27T06:14:03.525657+0000","last_unstale":"2024-03-27T06:22:15.428493+0000","last_undegraded":"2024-03-27T06:22:15.428493+0000","last_fullsized":"2024-03-27T06:22:15.428493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'465","last_scrub_stamp":"2024-03-27T06:22:15.428328+0000","last_deep_scrub":"44'465","last_deep_scrub_stamp":"2024-03-27T06:22:15.428328+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.428328+0000","objects_scrubbed":180,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:31:44.636833+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0.002460744,"stat_sum":{"num_bytes":18473602,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":23,"num_read_kb":12305,"num_write":340,"num_write_kb":22314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'592","reported_seq":667,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.564092+0000","last_change":"2024-03-27T06:22:23.564092+0000","last_active":"2024-03-27T06:22:23.564092+0000","last_peered":"2024-03-27T06:22:23.564092+0000","last_clean":"2024-03-27T06:22:23.564092+0000","last_became_active":"2024-03-27T06:14:02.535391+0000","last_became_peered":"2024-03-27T06:14:02.535391+0000","last_unstale":"2024-03-27T06:22:23.564092+0000","last_undegraded":"2024-03-27T06:22:23.564092+0000","last_fullsized":"2024-03-27T06:22:23.564092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'592","last_scrub_stamp":"2024-03-27T06:22:23.563803+0000","last_deep_scrub":"44'592","last_deep_scrub_stamp":"2024-03-27T06:22:23.563803+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.563803+0000","objects_scrubbed":6,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:47:41.195747+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388630,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":589,"num_write_kb":8551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30433,"num_omap_keys":53,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"44'439","reported_seq":2555,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:22.755158+0000","last_change":"2024-03-27T06:22:22.755158+0000","last_active":"2024-03-27T06:22:22.755158+0000","last_peered":"2024-03-27T06:22:22.755158+0000","last_clean":"2024-03-27T06:22:22.755158+0000","last_became_active":"2024-03-27T06:14:03.521090+0000","last_became_peered":"2024-03-27T06:14:03.521090+0000","last_unstale":"2024-03-27T06:22:22.755158+0000","last_undegraded":"2024-03-27T06:22:22.755158+0000","last_fullsized":"2024-03-27T06:22:22.755158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'439","last_scrub_stamp":"2024-03-27T06:22:22.755002+0000","last_deep_scrub":"44'439","last_deep_scrub_stamp":"2024-03-27T06:22:22.755002+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.755002+0000","objects_scrubbed":165,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:47:17.868267+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.0032966190000000002,"stat_sum":{"num_bytes":6064397,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":331,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"44'140","reported_seq":219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.925163+0000","last_change":"2024-03-27T06:22:19.925211+0000","last_active":"2024-03-27T06:22:19.925163+0000","last_peered":"2024-03-27T06:22:19.925163+0000","last_clean":"2024-03-27T06:22:19.925163+0000","last_became_active":"2024-03-27T06:14:02.535069+0000","last_became_peered":"2024-03-27T06:14:02.535069+0000","last_unstale":"2024-03-27T06:22:19.925163+0000","last_undegraded":"2024-03-27T06:22:19.925163+0000","last_fullsized":"2024-03-27T06:22:19.925163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'140","last_scrub_stamp":"2024-03-27T06:22:19.925054+0000","last_deep_scrub":"44'140","last_deep_scrub_stamp":"2024-03-27T06:22:19.925054+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.925054+0000","objects_scrubbed":2,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:47:15.038319+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":327,"num_write_kb":2013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"44'404","reported_seq":2519,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:24.023894+0000","last_change":"2024-03-27T06:22:24.023960+0000","last_active":"2024-03-27T06:22:24.023894+0000","last_peered":"2024-03-27T06:22:24.023894+0000","last_clean":"2024-03-27T06:22:24.023894+0000","last_became_active":"2024-03-27T06:14:03.528836+0000","last_became_peered":"2024-03-27T06:14:03.528836+0000","last_unstale":"2024-03-27T06:22:24.023894+0000","last_undegraded":"2024-03-27T06:22:24.023894+0000","last_fullsized":"2024-03-27T06:22:24.023894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'404","last_scrub_stamp":"2024-03-27T06:22:24.023833+0000","last_deep_scrub":"44'404","last_deep_scrub_stamp":"2024-03-27T06:22:24.023833+0000","last_clean_scrub_stamp":"2024-03-27T06:22:24.023833+0000","objects_scrubbed":147,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:52:11.609855+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0.0017608820000000001,"stat_sum":{"num_bytes":20225923,"num_objects":147,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":318,"num_write_kb":23984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"44'82","reported_seq":157,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.488229+0000","last_change":"2024-03-27T06:22:19.488259+0000","last_active":"2024-03-27T06:22:19.488229+0000","last_peered":"2024-03-27T06:22:19.488229+0000","last_clean":"2024-03-27T06:22:19.488229+0000","last_became_active":"2024-03-27T06:14:02.540399+0000","last_became_peered":"2024-03-27T06:14:02.540399+0000","last_unstale":"2024-03-27T06:22:19.488229+0000","last_undegraded":"2024-03-27T06:22:19.488229+0000","last_fullsized":"2024-03-27T06:22:19.488229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'82","last_scrub_stamp":"2024-03-27T06:22:19.488162+0000","last_deep_scrub":"44'82","last_deep_scrub_stamp":"2024-03-27T06:22:19.488162+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.488162+0000","objects_scrubbed":3,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:00:28.018114+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":78,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'470","reported_seq":2604,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:23.042179+0000","last_change":"2024-03-27T06:22:23.042237+0000","last_active":"2024-03-27T06:22:23.042179+0000","last_peered":"2024-03-27T06:22:23.042179+0000","last_clean":"2024-03-27T06:22:23.042179+0000","last_became_active":"2024-03-27T06:14:03.528639+0000","last_became_peered":"2024-03-27T06:14:03.528639+0000","last_unstale":"2024-03-27T06:22:23.042179+0000","last_undegraded":"2024-03-27T06:22:23.042179+0000","last_fullsized":"2024-03-27T06:22:23.042179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'470","last_scrub_stamp":"2024-03-27T06:22:23.042083+0000","last_deep_scrub":"44'470","last_deep_scrub_stamp":"2024-03-27T06:22:23.042083+0000","last_clean_scrub_stamp":"2024-03-27T06:22:23.042083+0000","objects_scrubbed":168,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:07:58.230056+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0.0018869889999999999,"stat_sum":{"num_bytes":32233430,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":35,"num_read_kb":18086,"num_write":345,"num_write_kb":31650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"44'35","reported_seq":110,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:13.054417+0000","last_change":"2024-03-27T06:22:13.054470+0000","last_active":"2024-03-27T06:22:13.054417+0000","last_peered":"2024-03-27T06:22:13.054417+0000","last_clean":"2024-03-27T06:22:13.054417+0000","last_became_active":"2024-03-27T06:14:02.527409+0000","last_became_peered":"2024-03-27T06:14:02.527409+0000","last_unstale":"2024-03-27T06:22:13.054417+0000","last_undegraded":"2024-03-27T06:22:13.054417+0000","last_fullsized":"2024-03-27T06:22:13.054417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'35","last_scrub_stamp":"2024-03-27T06:22:13.054293+0000","last_deep_scrub":"44'35","last_deep_scrub_stamp":"2024-03-27T06:22:13.054293+0000","last_clean_scrub_stamp":"2024-03-27T06:22:13.054293+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:14:29.311582+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":35,"num_write_kb":4230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27209,"num_omap_keys":47,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'456","reported_seq":2576,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.751632+0000","last_change":"2024-03-27T06:22:15.751711+0000","last_active":"2024-03-27T06:22:15.751632+0000","last_peered":"2024-03-27T06:22:15.751632+0000","last_clean":"2024-03-27T06:22:15.751632+0000","last_became_active":"2024-03-27T06:14:03.538960+0000","last_became_peered":"2024-03-27T06:14:03.538960+0000","last_unstale":"2024-03-27T06:22:15.751632+0000","last_undegraded":"2024-03-27T06:22:15.751632+0000","last_fullsized":"2024-03-27T06:22:15.751632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'456","last_scrub_stamp":"2024-03-27T06:22:15.751535+0000","last_deep_scrub":"44'456","last_deep_scrub_stamp":"2024-03-27T06:22:15.751535+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.751535+0000","objects_scrubbed":170,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:15:49.498938+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0.0081323179999999995,"stat_sum":{"num_bytes":22949808,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":346,"num_write_kb":22595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"44'125","reported_seq":200,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.827559+0000","last_change":"2024-03-27T06:22:17.827585+0000","last_active":"2024-03-27T06:22:17.827559+0000","last_peered":"2024-03-27T06:22:17.827559+0000","last_clean":"2024-03-27T06:22:17.827559+0000","last_became_active":"2024-03-27T06:14:02.533611+0000","last_became_peered":"2024-03-27T06:14:02.533611+0000","last_unstale":"2024-03-27T06:22:17.827559+0000","last_undegraded":"2024-03-27T06:22:17.827559+0000","last_fullsized":"2024-03-27T06:22:17.827559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'125","last_scrub_stamp":"2024-03-27T06:22:17.827494+0000","last_deep_scrub":"44'125","last_deep_scrub_stamp":"2024-03-27T06:22:17.827494+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.827494+0000","objects_scrubbed":6,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T16:59:17.962972+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":126,"num_write_kb":4511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"44'476","reported_seq":2598,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.036669+0000","last_change":"2024-03-27T06:22:21.036698+0000","last_active":"2024-03-27T06:22:21.036669+0000","last_peered":"2024-03-27T06:22:21.036669+0000","last_clean":"2024-03-27T06:22:21.036669+0000","last_became_active":"2024-03-27T06:14:03.524402+0000","last_became_peered":"2024-03-27T06:14:03.524402+0000","last_unstale":"2024-03-27T06:22:21.036669+0000","last_undegraded":"2024-03-27T06:22:21.036669+0000","last_fullsized":"2024-03-27T06:22:21.036669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'476","last_scrub_stamp":"2024-03-27T06:22:21.036606+0000","last_deep_scrub":"44'476","last_deep_scrub_stamp":"2024-03-27T06:22:21.036606+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.036606+0000","objects_scrubbed":185,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:54:03.408172+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0.001751964,"stat_sum":{"num_bytes":19413938,"num_objects":185,"num_object_clones":0,"num_object_copies":370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":21,"num_read_kb":12304,"num_write":357,"num_write_kb":23191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"44'28","reported_seq":102,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.718761+0000","last_change":"2024-03-27T06:22:17.718822+0000","last_active":"2024-03-27T06:22:17.718761+0000","last_peered":"2024-03-27T06:22:17.718761+0000","last_clean":"2024-03-27T06:22:17.718761+0000","last_became_active":"2024-03-27T06:14:02.517276+0000","last_became_peered":"2024-03-27T06:14:02.517276+0000","last_unstale":"2024-03-27T06:22:17.718761+0000","last_undegraded":"2024-03-27T06:22:17.718761+0000","last_fullsized":"2024-03-27T06:22:17.718761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'28","last_scrub_stamp":"2024-03-27T06:22:17.718655+0000","last_deep_scrub":"44'28","last_deep_scrub_stamp":"2024-03-27T06:22:17.718655+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.718655+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:05:05.339570+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":4098,"num_write":32,"num_write_kb":4123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13061,"num_omap_keys":23,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'435","reported_seq":2552,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:21.499080+0000","last_change":"2024-03-27T06:22:21.499133+0000","last_active":"2024-03-27T06:22:21.499080+0000","last_peered":"2024-03-27T06:22:21.499080+0000","last_clean":"2024-03-27T06:22:21.499080+0000","last_became_active":"2024-03-27T06:14:03.515184+0000","last_became_peered":"2024-03-27T06:14:03.515184+0000","last_unstale":"2024-03-27T06:22:21.499080+0000","last_undegraded":"2024-03-27T06:22:21.499080+0000","last_fullsized":"2024-03-27T06:22:21.499080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'435","last_scrub_stamp":"2024-03-27T06:22:21.499011+0000","last_deep_scrub":"44'435","last_deep_scrub_stamp":"2024-03-27T06:22:21.499011+0000","last_clean_scrub_stamp":"2024-03-27T06:22:21.499011+0000","objects_scrubbed":163,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T15:20:14.101708+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0.001518585,"stat_sum":{"num_bytes":15553910,"num_objects":163,"num_object_clones":0,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":17,"num_read_kb":5089,"num_write":319,"num_write_kb":15359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"44'350","reported_seq":427,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:28.539296+0000","last_change":"2024-03-27T06:22:22.565184+0000","last_active":"2024-03-27T06:22:28.539296+0000","last_peered":"2024-03-27T06:22:28.539296+0000","last_clean":"2024-03-27T06:22:28.539296+0000","last_became_active":"2024-03-27T06:14:02.530728+0000","last_became_peered":"2024-03-27T06:14:02.530728+0000","last_unstale":"2024-03-27T06:22:28.539296+0000","last_undegraded":"2024-03-27T06:22:28.539296+0000","last_fullsized":"2024-03-27T06:22:28.539296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'343","last_scrub_stamp":"2024-03-27T06:22:22.565067+0000","last_deep_scrub":"44'343","last_deep_scrub_stamp":"2024-03-27T06:22:22.565067+0000","last_clean_scrub_stamp":"2024-03-27T06:22:22.565067+0000","objects_scrubbed":6,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:12:49.712744+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10405501,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":343,"num_write_kb":10369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":29139,"num_omap_keys":51,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'468","reported_seq":2584,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:17.668612+0000","last_change":"2024-03-27T06:22:17.668675+0000","last_active":"2024-03-27T06:22:17.668612+0000","last_peered":"2024-03-27T06:22:17.668612+0000","last_clean":"2024-03-27T06:22:17.668612+0000","last_became_active":"2024-03-27T06:14:03.533213+0000","last_became_peered":"2024-03-27T06:14:03.533213+0000","last_unstale":"2024-03-27T06:22:17.668612+0000","last_undegraded":"2024-03-27T06:22:17.668612+0000","last_fullsized":"2024-03-27T06:22:17.668612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'468","last_scrub_stamp":"2024-03-27T06:22:17.668530+0000","last_deep_scrub":"44'468","last_deep_scrub_stamp":"2024-03-27T06:22:17.668530+0000","last_clean_scrub_stamp":"2024-03-27T06:22:17.668530+0000","objects_scrubbed":183,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:13:07.477640+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0.0075044719999999999,"stat_sum":{"num_bytes":10744528,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":15,"num_read_kb":4109,"num_write":347,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'484","reported_seq":2609,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:33.714644+0000","last_change":"2024-03-27T06:22:33.714682+0000","last_active":"2024-03-27T06:22:33.714644+0000","last_peered":"2024-03-27T06:22:33.714644+0000","last_clean":"2024-03-27T06:22:33.714644+0000","last_became_active":"2024-03-27T06:14:03.554375+0000","last_became_peered":"2024-03-27T06:14:03.554375+0000","last_unstale":"2024-03-27T06:22:33.714644+0000","last_undegraded":"2024-03-27T06:22:33.714644+0000","last_fullsized":"2024-03-27T06:22:33.714644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'484","last_scrub_stamp":"2024-03-27T06:22:33.714575+0000","last_deep_scrub":"44'484","last_deep_scrub_stamp":"2024-03-27T06:22:33.714575+0000","last_clean_scrub_stamp":"2024-03-27T06:22:33.714575+0000","objects_scrubbed":162,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T17:49:36.344230+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0.0086575760000000002,"stat_sum":{"num_bytes":8008268,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":25,"num_read_kb":4119,"num_write":307,"num_write_kb":12040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"44'24","reported_seq":98,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:16.003117+0000","last_change":"2024-03-27T06:22:16.003170+0000","last_active":"2024-03-27T06:22:16.003117+0000","last_peered":"2024-03-27T06:22:16.003117+0000","last_clean":"2024-03-27T06:22:16.003117+0000","last_became_active":"2024-03-27T06:14:02.532360+0000","last_became_peered":"2024-03-27T06:14:02.532360+0000","last_unstale":"2024-03-27T06:22:16.003117+0000","last_undegraded":"2024-03-27T06:22:16.003117+0000","last_fullsized":"2024-03-27T06:22:16.003117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'24","last_scrub_stamp":"2024-03-27T06:22:16.003007+0000","last_deep_scrub":"44'24","last_deep_scrub_stamp":"2024-03-27T06:22:16.003007+0000","last_clean_scrub_stamp":"2024-03-27T06:22:16.003007+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T14:08:43.854552+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4099,"num_write":22,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"44'444","reported_seq":2559,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:19.451105+0000","last_change":"2024-03-27T06:22:19.451146+0000","last_active":"2024-03-27T06:22:19.451105+0000","last_peered":"2024-03-27T06:22:19.451105+0000","last_clean":"2024-03-27T06:22:19.451105+0000","last_became_active":"2024-03-27T06:14:03.554353+0000","last_became_peered":"2024-03-27T06:14:03.554353+0000","last_unstale":"2024-03-27T06:22:19.451105+0000","last_undegraded":"2024-03-27T06:22:19.451105+0000","last_fullsized":"2024-03-27T06:22:19.451105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'444","last_scrub_stamp":"2024-03-27T06:22:19.451036+0000","last_deep_scrub":"44'444","last_deep_scrub_stamp":"2024-03-27T06:22:19.451036+0000","last_clean_scrub_stamp":"2024-03-27T06:22:19.451036+0000","objects_scrubbed":170,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T18:19:08.934850+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0.0061971559999999997,"stat_sum":{"num_bytes":27398907,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":15,"num_read_kb":8205,"num_write":334,"num_write_kb":31029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"44'60","reported_seq":139,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:15.036903+0000","last_change":"2024-03-27T06:22:15.036966+0000","last_active":"2024-03-27T06:22:15.036903+0000","last_peered":"2024-03-27T06:22:15.036903+0000","last_clean":"2024-03-27T06:22:15.036903+0000","last_became_active":"2024-03-27T06:14:02.533309+0000","last_became_peered":"2024-03-27T06:14:02.533309+0000","last_unstale":"2024-03-27T06:22:15.036903+0000","last_undegraded":"2024-03-27T06:22:15.036903+0000","last_fullsized":"2024-03-27T06:22:15.036903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'60","last_scrub_stamp":"2024-03-27T06:22:15.036766+0000","last_deep_scrub":"44'60","last_deep_scrub_stamp":"2024-03-27T06:22:15.036766+0000","last_clean_scrub_stamp":"2024-03-27T06:22:15.036766+0000","objects_scrubbed":5,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T13:00:16.842707+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":12312,"num_write":46,"num_write_kb":12316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"44'415","reported_seq":2536,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:27.189964+0000","last_change":"2024-03-27T06:22:27.190031+0000","last_active":"2024-03-27T06:22:27.189964+0000","last_peered":"2024-03-27T06:22:27.189964+0000","last_clean":"2024-03-27T06:22:27.189964+0000","last_became_active":"2024-03-27T06:14:03.530135+0000","last_became_peered":"2024-03-27T06:14:03.530135+0000","last_unstale":"2024-03-27T06:22:27.189964+0000","last_undegraded":"2024-03-27T06:22:27.189964+0000","last_fullsized":"2024-03-27T06:22:27.189964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'415","last_scrub_stamp":"2024-03-27T06:22:27.189868+0000","last_deep_scrub":"44'415","last_deep_scrub_stamp":"2024-03-27T06:22:27.189868+0000","last_clean_scrub_stamp":"2024-03-27T06:22:27.189868+0000","objects_scrubbed":145,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T11:47:44.821812+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0.0036426230000000002,"stat_sum":{"num_bytes":14422279,"num_objects":145,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":21,"num_read_kb":4923,"num_write":311,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"44'82","reported_seq":156,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.100162+0000","last_change":"2024-03-27T06:22:18.100206+0000","last_active":"2024-03-27T06:22:18.100162+0000","last_peered":"2024-03-27T06:22:18.100162+0000","last_clean":"2024-03-27T06:22:18.100162+0000","last_became_active":"2024-03-27T06:14:02.522904+0000","last_became_peered":"2024-03-27T06:14:02.522904+0000","last_unstale":"2024-03-27T06:22:18.100162+0000","last_undegraded":"2024-03-27T06:22:18.100162+0000","last_fullsized":"2024-03-27T06:22:18.100162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'82","last_scrub_stamp":"2024-03-27T06:22:18.100093+0000","last_deep_scrub":"44'82","last_deep_scrub_stamp":"2024-03-27T06:22:18.100093+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.100093+0000","objects_scrubbed":2,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:34:11.041505+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":75,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17639,"num_omap_keys":31,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"44'470","reported_seq":2585,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:28.196352+0000","last_change":"2024-03-27T06:22:28.196383+0000","last_active":"2024-03-27T06:22:28.196352+0000","last_peered":"2024-03-27T06:22:28.196352+0000","last_clean":"2024-03-27T06:22:28.196352+0000","last_became_active":"2024-03-27T06:14:03.531122+0000","last_became_peered":"2024-03-27T06:14:03.531122+0000","last_unstale":"2024-03-27T06:22:28.196352+0000","last_undegraded":"2024-03-27T06:22:28.196352+0000","last_fullsized":"2024-03-27T06:22:28.196352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'470","last_scrub_stamp":"2024-03-27T06:22:28.196287+0000","last_deep_scrub":"44'470","last_deep_scrub_stamp":"2024-03-27T06:22:28.196287+0000","last_clean_scrub_stamp":"2024-03-27T06:22:28.196287+0000","objects_scrubbed":166,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:26:38.909943+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0.0036711629999999999,"stat_sum":{"num_bytes":15186793,"num_objects":166,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":14,"num_read_kb":8204,"num_write":343,"num_write_kb":14950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"44'137","reported_seq":225,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-27T06:22:18.911720+0000","last_change":"2024-03-27T06:22:18.911720+0000","last_active":"2024-03-27T06:22:18.911720+0000","last_peered":"2024-03-27T06:22:18.911720+0000","last_clean":"2024-03-27T06:22:18.911720+0000","last_became_active":"2024-03-27T06:14:02.533923+0000","last_became_peered":"2024-03-27T06:14:02.533923+0000","last_unstale":"2024-03-27T06:22:18.911720+0000","last_undegraded":"2024-03-27T06:22:18.911720+0000","last_fullsized":"2024-03-27T06:22:18.911720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'137","last_scrub_stamp":"2024-03-27T06:22:18.911624+0000","last_deep_scrub":"44'137","last_deep_scrub_stamp":"2024-03-27T06:22:18.911624+0000","last_clean_scrub_stamp":"2024-03-27T06:22:18.911624+0000","objects_scrubbed":10,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T12:29:46.234836+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":4136,"num_write":131,"num_write_kb":8314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":52935,"num_omap_keys":92,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":940893603,"num_objects":10589,"num_object_clones":0,"num_object_copies":21178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10589,"num_whiteouts":0,"num_read":1256,"num_read_kb":394723,"num_write":21174,"num_write_kb":1064721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1712472064,"data_stored":1880247110,"data_compressed":164906076,"data_compressed_allocated":182468608,"data_compressed_original":405446656,"omap_allocated":0,"internal_metadata":0},"log_size":28882,"ondisk_log_size":28882,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":182911582,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":453,"num_read_kb":120084,"num_write":4130,"num_write_kb":184163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":97,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706809,"num_omap_keys":1217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81707008,"data_stored":365823164,"data_compressed":49818618,"data_compressed_allocated":62414848,"data_compressed_original":346685440,"omap_allocated":0,"internal_metadata":0},"log_size":4064,"ondisk_log_size":4064,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64738,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574955,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192748,"kb_used_data":143524,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94179092,"statfs":{"total":96636764160,"available":96439390208,"internally_reserved":0,"allocated":146968576,"data_stored":174700599,"data_compressed":11348325,"data_compressed_allocated":13275136,"data_compressed_original":46198784,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214948,"kb_used_data":169884,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94156892,"statfs":{"total":96636764160,"available":96416657408,"internally_reserved":0,"allocated":173961216,"data_stored":215859568,"data_compressed":22390241,"data_compressed_allocated":25247744,"data_compressed_original":73641984,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574956,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296068,"kb_used_data":242684,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94075772,"statfs":{"total":96636764160,"available":96333590528,"internally_reserved":0,"allocated":248508416,"data_stored":304860587,"data_compressed":28268738,"data_compressed_allocated":32043008,"data_compressed_original":96636928,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270932,"kb_used_data":217548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94100908,"statfs":{"total":96636764160,"available":96359329792,"internally_reserved":0,"allocated":222769152,"data_stored":282203639,"data_compressed":27135224,"data_compressed_allocated":30642176,"data_compressed_original":95821824,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336452,"kb_used_data":265020,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94035388,"statfs":{"total":96636764160,"available":96292237312,"internally_reserved":0,"allocated":271380480,"data_stored":359685749,"data_compressed":39583813,"data_compressed_allocated":45559808,"data_compressed_original":142385152,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271896,"kb_used_data":222672,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94099944,"statfs":{"total":96636764160,"available":96358342656,"internally_reserved":0,"allocated":228016128,"data_stored":275897170,"data_compressed":26602197,"data_compressed_allocated":30121984,"data_compressed_original":85983232,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":251104,"kb_used_data":201880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94120736,"statfs":{"total":96636764160,"available":96379633664,"internally_reserved":0,"allocated":206725120,"data_stored":255586545,"data_compressed":23288424,"data_compressed_allocated":26611712,"data_compressed_original":81715200,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364380,"kb_used_data":292948,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94007460,"statfs":{"total":96636764160,"available":96263639040,"internally_reserved":0,"allocated":299978752,"data_stored":380226993,"data_compressed":36172470,"data_compressed_allocated":41472000,"data_compressed_original":130895872,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32369,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":67110807,"data_compressed":8915435,"data_compressed_allocated":11333632,"data_compressed_original":64311296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7794688,"data_stored":37748896,"data_compressed":4858381,"data_compressed_allocated":6283264,"data_compressed_original":36245504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7151616,"data_stored":31377727,"data_compressed":4492512,"data_compressed_allocated":5570560,"data_compressed_original":29810688,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19951616,"data_stored":81710522,"data_compressed":11199248,"data_compressed_allocated":14315520,"data_compressed_original":76091392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":50333888,"data_compressed":7205527,"data_compressed_allocated":8712192,"data_compressed_original":46735360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8495104,"data_stored":42478552,"data_compressed":5618062,"data_compressed_allocated":6979584,"data_compressed_original":40984576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5763072,"data_stored":29361843,"data_compressed":4140996,"data_compressed_allocated":5042176,"data_compressed_original":28651520,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":25700929,"data_compressed":3388457,"data_compressed_allocated":4177920,"data_compressed_original":23855104,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285327360,"data_stored":312894956,"data_compressed":27257035,"data_compressed_allocated":30138368,"data_compressed_original":66584576,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198365184,"data_stored":217026051,"data_compressed":18397674,"data_compressed_allocated":20283392,"data_compressed_original":44896256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220364800,"data_stored":244298213,"data_compressed":22109685,"data_compressed_allocated":24551424,"data_compressed_original":56172544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250929152,"data_stored":277753997,"data_compressed":28384565,"data_compressed_allocated":31244288,"data_compressed_original":66293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209920000,"data_stored":231648521,"data_compressed":19929697,"data_compressed_allocated":21929984,"data_compressed_original":49086464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239448064,"data_stored":261570437,"data_compressed":22618307,"data_compressed_allocated":25018368,"data_compressed_original":55078912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167698432,"data_stored":186276495,"data_compressed":18249245,"data_compressed_allocated":20205568,"data_compressed_original":44990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140419072,"data_stored":148778440,"data_compressed":7959868,"data_compressed_allocated":9097216,"data_compressed_original":22343680,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T06:22:38.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-27T06:22:39.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.240+0000 7f4aafc00640 1 Processor -- start 2024-03-27T06:22:39.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.241+0000 7f4aafc00640 1 -- start start 2024-03-27T06:22:39.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aafc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f4aa815bfa0 0x7f4aa815c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aafc00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4aa815c380 0x7f4aa80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aafc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa8071930 con 0x7f4aa8152500 2024-03-27T06:22:39.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aafc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa8072110 con 0x7f4aa815bfa0 2024-03-27T06:22:39.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aafc00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa806c530 con 0x7f4aa815c380 2024-03-27T06:22:39.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4a9fe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4aa815c380 0x7f4aa80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4a9fe00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4aa815c380 0x7f4aa80735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54028/0 (socket says 172.21.15.132:54028) 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4a9fe00640 1 -- 172.21.15.132:0/3196291458 learned_addr learned my addr 172.21.15.132:0/3196291458 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.242+0000 7f4aad200640 1 --2- 172.21.15.132:0/3196291458 >> v2:172.21.15.132:3300/0 conn(0x7f4aa815bfa0 0x7f4aa815c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.243+0000 7f4a9fe00640 1 -- 172.21.15.132:0/3196291458 >> v2:172.21.15.132:3300/0 conn(0x7f4aa815bfa0 msgr2=0x7f4aa815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.243+0000 7f4a9fe00640 1 --2- 172.21.15.132:0/3196291458 >> v2:172.21.15.132:3300/0 conn(0x7f4aa815bfa0 0x7f4aa815c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.243+0000 7f4a9fe00640 1 -- 172.21.15.132:0/3196291458 >> v1:172.21.15.132:6789/0 conn(0x7f4aa8152500 legacy=0x7f4aa806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.243+0000 7f4a9fe00640 1 -- 172.21.15.132:0/3196291458 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa8071bd0 con 0x7f4aa815c380 2024-03-27T06:22:39.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4a9fe00640 1 --2- 172.21.15.132:0/3196291458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa80735f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4aa806fdb0 tx=0x7f4a9002ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee11eca3fd84d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4aa7e00640 1 -- 172.21.15.132:0/3196291458 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9002f850 con 0x7f4aa815c380 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4aa7e00640 1 -- 172.21.15.132:0/3196291458 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a9002f9f0 con 0x7f4aa815c380 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4aa7e00640 1 -- 172.21.15.132:0/3196291458 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9002fdc0 con 0x7f4aa815c380 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4aafc00640 1 -- 172.21.15.132:0/3196291458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 msgr2=0x7f4aa80735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.244+0000 7f4aafc00640 1 --2- 172.21.15.132:0/3196291458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa80735f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4aa806fdb0 tx=0x7f4a9002ba30 comp rx=0 tx=0).stop 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 -- 172.21.15.132:0/3196291458 shutdown_connections 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 --2- 172.21.15.132:0/3196291458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa80735f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 --2- 172.21.15.132:0/3196291458 >> v2:172.21.15.132:3300/0 conn(0x7f4aa815bfa0 0x7f4aa815c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 -- 172.21.15.132:0/3196291458 >> 172.21.15.132:0/3196291458 conn(0x7f4aa80c89a0 msgr2=0x7f4aa80c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:39.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 -- 172.21.15.132:0/3196291458 shutdown_connections 2024-03-27T06:22:39.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.245+0000 7f4aafc00640 1 -- 172.21.15.132:0/3196291458 wait complete. 2024-03-27T06:22:39.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.246+0000 7f4aafc00640 1 Processor -- start 2024-03-27T06:22:39.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 -- start start 2024-03-27T06:22:39.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 0x7f4aa81e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa814b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa8071e70 con 0x7f4aa8152500 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa80c97e0 con 0x7f4aa815c380 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.247+0000 7f4aafc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa806ca60 con 0x7f4aa815bfa0 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4aad200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4aadc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa814b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 0x7f4aa81e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4aad200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37592/0 (socket says 172.21.15.132:37592) 2024-03-27T06:22:39.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4aad200640 1 -- 172.21.15.132:0/1421817267 learned_addr learned my addr 172.21.15.132:0/1421817267 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 -- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 msgr2=0x7f4aa814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa814b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 -- 172.21.15.132:0/1421817267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 msgr2=0x7f4aa81e4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4a9fe00640 1 -- 172.21.15.132:0/1421817267 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa81f2660 con 0x7f4aa815bfa0 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.248+0000 7f4aad200640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:39.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4a9fe00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 0x7f4aa81e1f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4aa806c6b0 tx=0x7f4a90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90046070 con 0x7f4aa815bfa0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a90037be0 con 0x7f4aa815bfa0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90041440 con 0x7f4aa815bfa0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa81ee6e0 con 0x7f4aa815bfa0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.249+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa81eeca0 con 0x7f4aa815bfa0 2024-03-27T06:22:39.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.250+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa8068950 con 0x7f4aa815bfa0 2024-03-27T06:22:39.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.251+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f4a90049070 con 0x7f4aa815bfa0 2024-03-27T06:22:39.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.252+0000 7f4aa5600640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 0x7f4a8406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.252+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f4a900ad740 con 0x7f4aa815bfa0 2024-03-27T06:22:39.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.252+0000 7f4aad200640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 0x7f4a8406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.253+0000 7f4aad200640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 0x7f4a8406e9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4a980112c0 tx=0x7f4a98009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.257+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f4a90033030 con 0x7f4aa815bfa0 2024-03-27T06:22:39.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.457+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4aa814fef0 con 0x7f4aa815bfa0 2024-03-27T06:22:39.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.464+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4a9007b450 con 0x7f4aa815bfa0 2024-03-27T06:22:39.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.464+0000 7f4aa5600640 1 -- 172.21.15.132:0/1421817267 <== mon.2 v2:172.21.15.134:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f4a9007f750 con 0x7f4aa815bfa0 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 msgr2=0x7f4a8406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 0x7f4a8406e9f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4a980112c0 tx=0x7f4a98009190 comp rx=0 tx=0).stop 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 msgr2=0x7f4aa81e1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 0x7f4aa81e1f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4aa806c6b0 tx=0x7f4a90004730 comp rx=0 tx=0).stop 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 shutdown_connections 2024-03-27T06:22:39.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f4a8406c530 0x7f4a8406e9f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4aa815c380 0x7f4aa814b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f4aa815bfa0 0x7f4aa81e1f30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 --2- 172.21.15.132:0/1421817267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4aa8152500 0x7f4aa81e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 >> 172.21.15.132:0/1421817267 conn(0x7f4aa80c89a0 msgr2=0x7f4aa806e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 shutdown_connections 2024-03-27T06:22:39.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.467+0000 7f4aafc00640 1 -- 172.21.15.132:0/1421817267 wait complete. 2024-03-27T06:22:39.474 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-27T06:22:39.474 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:22:39.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.687+0000 7f2d36a00640 1 Processor -- start 2024-03-27T06:22:39.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.687+0000 7f2d36a00640 1 -- start start 2024-03-27T06:22:39.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2d300cd6a0 0x7f2d3006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d3015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3015de10 con 0x7f2d301549a0 2024-03-27T06:22:39.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f2d300cada0 con 0x7f2d300cd6a0 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2d300cdad0 con 0x7f2d3006be10 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d35600640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2d301549a0 0x7f2d300cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:47458/0 (socket says 172.21.15.132:47458) 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d35600640 1 -- 172.21.15.132:0/2937766062 learned_addr learned my addr 172.21.15.132:0/2937766062 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2937766062 >> v2:172.21.15.134:3300/0 conn(0x7f2d300cd6a0 0x7f2d3006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.690+0000 7f2d36000640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d3015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d2fe00640 1 -- 172.21.15.132:0/2937766062 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906896309 0 0) 0x7f2d3015de10 con 0x7f2d301549a0 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d2fe00640 1 -- 172.21.15.132:0/2937766062 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d18003520 con 0x7f2d301549a0 2024-03-27T06:22:39.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d34c00640 1 -- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 msgr2=0x7f2d3015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:39.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d3015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d34c00640 1 -- 172.21.15.132:0/2937766062 >> v1:172.21.15.134:6789/0 conn(0x7f2d301549a0 legacy=0x7f2d300cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.691+0000 7f2d34c00640 1 -- 172.21.15.132:0/2937766062 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3015e0b0 con 0x7f2d300cd6a0 2024-03-27T06:22:39.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d300cd6a0 0x7f2d3006b8d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0097a0 tx=0x7f2d1c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5d5218a9fc95fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d2fe00640 1 -- 172.21.15.132:0/2937766062 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c034070 con 0x7f2d300cd6a0 2024-03-27T06:22:39.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d2fe00640 1 -- 172.21.15.132:0/2937766062 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2d1c030610 con 0x7f2d300cd6a0 2024-03-27T06:22:39.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d2fe00640 1 -- 172.21.15.132:0/2937766062 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c030a00 con 0x7f2d300cd6a0 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d36a00640 1 -- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d300cd6a0 msgr2=0x7f2d3006b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.693+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d300cd6a0 0x7f2d3006b8d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0097a0 tx=0x7f2d1c02b960 comp rx=0 tx=0).stop 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 -- 172.21.15.132:0/2937766062 shutdown_connections 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d3015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2937766062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d300cd6a0 0x7f2d3006b8d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 -- 172.21.15.132:0/2937766062 >> 172.21.15.132:0/2937766062 conn(0x7f2d3006d8d0 msgr2=0x7f2d3006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 -- 172.21.15.132:0/2937766062 shutdown_connections 2024-03-27T06:22:39.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.694+0000 7f2d36a00640 1 -- 172.21.15.132:0/2937766062 wait complete. 2024-03-27T06:22:39.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.695+0000 7f2d36a00640 1 Processor -- start 2024-03-27T06:22:39.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.695+0000 7f2d36a00640 1 -- start start 2024-03-27T06:22:39.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d300cd6a0 0x7f2d30153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d301549a0 0x7f2d300cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3015e950 con 0x7f2d300cd6a0 2024-03-27T06:22:39.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3015e670 con 0x7f2d301549a0 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.696+0000 7f2d36a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3015ec30 con 0x7f2d3006be10 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50458/0 (socket says 172.21.15.132:50458) 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d36000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d301549a0 0x7f2d300cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 -- 172.21.15.132:0/2695286839 learned_addr learned my addr 172.21.15.132:0/2695286839 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d300cd6a0 0x7f2d30153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 -- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d301549a0 msgr2=0x7f2d300cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d301549a0 0x7f2d300cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 -- 172.21.15.132:0/2695286839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d300cd6a0 msgr2=0x7f2d30153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d300cd6a0 0x7f2d30153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 -- 172.21.15.132:0/2695286839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d300ce460 con 0x7f2d3006be10 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d35600640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c410 tx=0x7f2d2400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24017070 con 0x7f2d3006be10 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2d24005ce0 con 0x7f2d3006be10 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.697+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24007310 con 0x7f2d3006be10 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.698+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d300ce740 con 0x7f2d3006be10 2024-03-27T06:22:39.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.698+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d300cf790 con 0x7f2d3006be10 2024-03-27T06:22:39.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.699+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d300cec60 con 0x7f2d3006be10 2024-03-27T06:22:39.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.699+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f2d2401a070 con 0x7f2d3006be10 2024-03-27T06:22:39.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.700+0000 7f2d2d600640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 0x7f2d1006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:39.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.700+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f2d2408ae60 con 0x7f2d3006be10 2024-03-27T06:22:39.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.700+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 0x7f2d1006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:39.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.701+0000 7f2d34c00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 0x7f2d1006e9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c036000 tx=0x7f2d1c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:39.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.704+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2d300cec60 con 0x7f2d3006be10 2024-03-27T06:22:39.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.946+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d300686c0 con 0x7f2d3006be10 2024-03-27T06:22:39.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.947+0000 7f2d2d600640 1 -- 172.21.15.132:0/2695286839 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6890 (secure 0 0 0) 0x7f2d24058bf0 con 0x7f2d3006be10 2024-03-27T06:22:39.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:39.937 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5111,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1016263278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1016263278},{"type":"v1","addr":"172.21.15.132:6837","nonce":1016263278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:50.235819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:active","state_seq":36,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[5284,4904,4817]},"id":1}]} 2024-03-27T06:22:39.937 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 42 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 msgr2=0x7f2d1006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 0x7f2d1006e9f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c036000 tx=0x7f2d1c03b040 comp rx=0 tx=0).stop 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 msgr2=0x7f2d301f0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c410 tx=0x7f2d2400c8d0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 shutdown_connections 2024-03-27T06:22:39.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f2d1006c530 0x7f2d1006e9f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d301549a0 0x7f2d300cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d300cd6a0 0x7f2d30153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 --2- 172.21.15.132:0/2695286839 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f2d3006be10 0x7f2d301f0ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 >> 172.21.15.132:0/2695286839 conn(0x7f2d3006d8d0 msgr2=0x7f2d301583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 shutdown_connections 2024-03-27T06:22:39.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:39.950+0000 7f2d36a00640 1 -- 172.21.15.132:0/2695286839 wait complete. 2024-03-27T06:22:39.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-27T06:22:40.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.205+0000 7f9f67a00640 1 Processor -- start 2024-03-27T06:22:40.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.205+0000 7f9f67a00640 1 -- start start 2024-03-27T06:22:40.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f67a00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9f60152950 0x7f9f601590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f67a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9f60073970 0x7f9f6006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f67a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f6006b630 con 0x7f9f60073550 2024-03-27T06:22:40.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f67a00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f9f60152d30 con 0x7f9f60152950 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f67a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f9f6006b8d0 con 0x7f9f60073970 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f57e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9f60073550 0x7f9f60071170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:47466/0 (socket says 172.21.15.132:47466) 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f65000640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f9f60152950 0x7f9f601590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f57e00640 1 -- 172.21.15.132:0/2118481425 learned_addr learned my addr 172.21.15.132:0/2118481425 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.206+0000 7f9f65a00640 1 --2- 172.21.15.132:0/2118481425 >> v2:172.21.15.134:3300/0 conn(0x7f9f60073970 0x7f9f6006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f5fe00640 1 -- 172.21.15.132:0/2118481425 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954509060 0 0) 0x7f9f6006b630 con 0x7f9f60073550 2024-03-27T06:22:40.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f5fe00640 1 -- 172.21.15.132:0/2118481425 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f44003520 con 0x7f9f60073550 2024-03-27T06:22:40.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f65a00640 1 -- 172.21.15.132:0/2118481425 >> v2:172.21.15.132:3300/0 conn(0x7f9f60152950 msgr2=0x7f9f601590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:40.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f65a00640 1 --2- 172.21.15.132:0/2118481425 >> v2:172.21.15.132:3300/0 conn(0x7f9f60152950 0x7f9f601590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f65a00640 1 -- 172.21.15.132:0/2118481425 >> v1:172.21.15.134:6789/0 conn(0x7f9f60073550 legacy=0x7f9f60071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.207+0000 7f9f65a00640 1 -- 172.21.15.132:0/2118481425 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6006bbb0 con 0x7f9f60073970 2024-03-27T06:22:40.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.208+0000 7f9f65a00640 1 --2- 172.21.15.132:0/2118481425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073970 0x7f9f6006cf50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009d90 tx=0x7f9f5802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f10b3d6870ce0e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.208+0000 7f9f5fe00640 1 -- 172.21.15.132:0/2118481425 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58039070 con 0x7f9f60073970 2024-03-27T06:22:40.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.208+0000 7f9f5fe00640 1 -- 172.21.15.132:0/2118481425 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f9f5802cc60 con 0x7f9f60073970 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.208+0000 7f9f5fe00640 1 -- 172.21.15.132:0/2118481425 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58034680 con 0x7f9f60073970 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 -- 172.21.15.132:0/2118481425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073970 msgr2=0x7f9f6006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 --2- 172.21.15.132:0/2118481425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073970 0x7f9f6006cf50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009d90 tx=0x7f9f5802beb0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 -- 172.21.15.132:0/2118481425 shutdown_connections 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 --2- 172.21.15.132:0/2118481425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073970 0x7f9f6006cf50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 --2- 172.21.15.132:0/2118481425 >> v2:172.21.15.132:3300/0 conn(0x7f9f60152950 0x7f9f601590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 -- 172.21.15.132:0/2118481425 >> 172.21.15.132:0/2118481425 conn(0x7f9f600cc060 msgr2=0x7f9f600cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:40.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 -- 172.21.15.132:0/2118481425 shutdown_connections 2024-03-27T06:22:40.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.209+0000 7f9f67a00640 1 -- 172.21.15.132:0/2118481425 wait complete. 2024-03-27T06:22:40.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.210+0000 7f9f67a00640 1 Processor -- start 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.211+0000 7f9f67a00640 1 -- start start 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.211+0000 7f9f67a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.211+0000 7f9f67a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 0x7f9f601f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f67a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f60152950 0x7f9f601f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f67a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6006c450 con 0x7f9f60073970 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f67a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6006c170 con 0x7f9f60073550 2024-03-27T06:22:40.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f67a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f9f60071880 con 0x7f9f60152950 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f57e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 0x7f9f601f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54074/0 (socket says 172.21.15.132:54074) 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 -- 172.21.15.132:0/3080569925 learned_addr learned my addr 172.21.15.132:0/3080569925 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f60152950 0x7f9f601f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 -- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f60152950 msgr2=0x7f9f601f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f60152950 0x7f9f601f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 -- 172.21.15.132:0/3080569925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 msgr2=0x7f9f601f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 0x7f9f601f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 -- 172.21.15.132:0/3080569925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f60151c20 con 0x7f9f60073550 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f57e00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 0x7f9f601f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.212+0000 7f9f65000640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e8a0 tx=0x7f9f5000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.213+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f50017070 con 0x7f9f60073550 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.213+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f9f5000c940 con 0x7f9f60073550 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.213+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f60159f80 con 0x7f9f60073550 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.213+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000cd50 con 0x7f9f60073550 2024-03-27T06:22:40.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.213+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6015c900 con 0x7f9f60073550 2024-03-27T06:22:40.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.214+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6015a150 con 0x7f9f60073550 2024-03-27T06:22:40.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.215+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f9f5001a070 con 0x7f9f60073550 2024-03-27T06:22:40.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.216+0000 7f9f5d600640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 0x7f9f3c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.216+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f9f5008bc20 con 0x7f9f60073550 2024-03-27T06:22:40.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.216+0000 7f9f57e00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 0x7f9f3c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.216+0000 7f9f57e00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 0x7f9f3c06e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f600ca590 tx=0x7f9f48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.220+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f9f500598f0 con 0x7f9f60073550 2024-03-27T06:22:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.477+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f60068840 con 0x7f9f60073550 2024-03-27T06:22:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.478+0000 7f9f5d600640 1 -- 172.21.15.132:0/3080569925 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6890 (secure 0 0 0) 0x7f9f5005dbf0 con 0x7f9f60073550 2024-03-27T06:22:40.467 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:40.468 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4272768803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4272768803},{"type":"v1","addr":"172.21.15.134:6835","nonce":4272768803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/3378771917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":3378771917},{"type":"v1","addr":"172.21.15.132:6833","nonce":3378771917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5111,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/1016263278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6835","nonce":1016263278},{"type":"v1","addr":"172.21.15.132:6837","nonce":1016263278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-27T06:14:02.985226+0000","modified":"2024-03-27T06:21:50.235819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4904,"mds_1":4817,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4817":{"gid":4817,"name":"e","rank":1,"incarnation":25,"state":"up:active","state_seq":24,"addr":"172.21.15.132:6836/2917670201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":2917670201},{"type":"v1","addr":"172.21.15.132:6836","nonce":2917670201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4904":{"gid":4904,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":18,"addr":"172.21.15.134:6838/2682370774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2682370774},{"type":"v1","addr":"172.21.15.134:6838","nonce":2682370774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5284":{"gid":5284,"name":"f","rank":2,"incarnation":37,"state":"up:active","state_seq":36,"addr":"172.21.15.134:6839/4090015809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6837","nonce":4090015809},{"type":"v1","addr":"172.21.15.134:6839","nonce":4090015809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4904,"qdb_cluster":[5284,4904,4817]},"id":1}]} 2024-03-27T06:22:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:dumped fsmap epoch 42 2024-03-27T06:22:40.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 msgr2=0x7f9f3c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 0x7f9f3c06e9f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f600ca590 tx=0x7f9f48009040 comp rx=0 tx=0).stop 2024-03-27T06:22:40.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 msgr2=0x7f9f601e7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e8a0 tx=0x7f9f5000ed60 comp rx=0 tx=0).stop 2024-03-27T06:22:40.470 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 shutdown_connections 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.480+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f9f60152950 0x7f9f601f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f60073970 0x7f9f601f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f9f3c06c530 0x7f9f3c06e9f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 --2- 172.21.15.132:0/3080569925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f60073550 0x7f9f601e7270 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 >> 172.21.15.132:0/3080569925 conn(0x7f9f600cc060 msgr2=0x7f9f600c13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 shutdown_connections 2024-03-27T06:22:40.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.481+0000 7f9f67a00640 1 -- 172.21.15.132:0/3080569925 wait complete. 2024-03-27T06:22:40.489 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T06:22:40.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.715+0000 7f624d800640 1 Processor -- start 2024-03-27T06:22:40.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.716+0000 7f624d800640 1 -- start start 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f624d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f624d800640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f624806c3d0 0x7f62480c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f624d800640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624806ba60 con 0x7f624815bf30 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f624d800640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7f624815ba80 con 0x7f624806c3d0 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f624d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f624806bd40 con 0x7f6248154e00 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f6246a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f6246a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50478/0 (socket says 172.21.15.132:50478) 2024-03-27T06:22:40.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f6247400640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7f624806c3d0 0x7f62480c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.717+0000 7f6246a00640 1 -- 172.21.15.132:0/3903807653 learned_addr learned my addr 172.21.15.132:0/3903807653 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:40.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.718+0000 7f6246a00640 1 -- 172.21.15.132:0/3903807653 >> v2:172.21.15.132:3300/0 conn(0x7f624806c3d0 msgr2=0x7f62480c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.718+0000 7f6246a00640 1 --2- 172.21.15.132:0/3903807653 >> v2:172.21.15.132:3300/0 conn(0x7f624806c3d0 0x7f62480c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.718+0000 7f6246a00640 1 -- 172.21.15.132:0/3903807653 >> v1:172.21.15.134:6789/0 conn(0x7f624815bf30 legacy=0x7f62480c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.718+0000 7f6246a00640 1 -- 172.21.15.132:0/3903807653 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62480c8230 con 0x7f6248154e00 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.719+0000 7f6246a00640 1 --2- 172.21.15.132:0/3903807653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6238009870 tx=0x7f623802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb08b80d26b73b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.719+0000 7f6246000640 1 -- 172.21.15.132:0/3903807653 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238034070 con 0x7f6248154e00 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.719+0000 7f6246000640 1 -- 172.21.15.132:0/3903807653 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6238030840 con 0x7f6248154e00 2024-03-27T06:22:40.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.719+0000 7f6246000640 1 -- 172.21.15.132:0/3903807653 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238030c30 con 0x7f6248154e00 2024-03-27T06:22:40.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 -- 172.21.15.132:0/3903807653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 msgr2=0x7f624815b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 --2- 172.21.15.132:0/3903807653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6238009870 tx=0x7f623802b980 comp rx=0 tx=0).stop 2024-03-27T06:22:40.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 -- 172.21.15.132:0/3903807653 shutdown_connections 2024-03-27T06:22:40.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 --2- 172.21.15.132:0/3903807653 >> v2:172.21.15.132:3300/0 conn(0x7f624806c3d0 0x7f62480c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 --2- 172.21.15.132:0/3903807653 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f6248154e00 0x7f624815b540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 -- 172.21.15.132:0/3903807653 >> 172.21.15.132:0/3903807653 conn(0x7f62480d0150 msgr2=0x7f62480d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:40.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 -- 172.21.15.132:0/3903807653 shutdown_connections 2024-03-27T06:22:40.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.720+0000 7f624d800640 1 -- 172.21.15.132:0/3903807653 wait complete. 2024-03-27T06:22:40.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.722+0000 7f624d800640 1 Processor -- start 2024-03-27T06:22:40.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.722+0000 7f624d800640 1 -- start start 2024-03-27T06:22:40.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6248154e00 0x7f624814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f624815bf30 0x7f62481f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f62481551e0 con 0x7f6248154e00 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f62480c87f0 con 0x7f624815bf30 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f624d800640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f624806b520 con 0x7f624806c3d0 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50486/0 (socket says 172.21.15.132:50486) 2024-03-27T06:22:40.715 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 -- 172.21.15.132:0/2358842507 learned_addr learned my addr 172.21.15.132:0/2358842507 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6247400640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f624815bf30 0x7f62481f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f623e000640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6248154e00 0x7f624814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 -- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f624815bf30 msgr2=0x7f62481f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f624815bf30 0x7f62481f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 -- 172.21.15.132:0/2358842507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6248154e00 msgr2=0x7f624814c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6248154e00 0x7f624814c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.724+0000 7f6246a00640 1 -- 172.21.15.132:0/2358842507 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6248159cd0 con 0x7f624806c3d0 2024-03-27T06:22:40.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f6246a00640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f623802b9c0 tx=0x7f6238004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238048070 con 0x7f624806c3d0 2024-03-27T06:22:40.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6238030640 con 0x7f624806c3d0 2024-03-27T06:22:40.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238034040 con 0x7f624806c3d0 2024-03-27T06:22:40.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6248152c50 con 0x7f624806c3d0 2024-03-27T06:22:40.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.725+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6248153c80 con 0x7f624806c3d0 2024-03-27T06:22:40.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.727+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6248068fe0 con 0x7f624806c3d0 2024-03-27T06:22:40.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.727+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f6238037070 con 0x7f624806c3d0 2024-03-27T06:22:40.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.728+0000 7f623f400640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 0x7f622406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:40.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.728+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7f62380ad640 con 0x7f624806c3d0 2024-03-27T06:22:40.718 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.728+0000 7f623e000640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 0x7f622406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:40.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.728+0000 7f623e000640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 0x7f622406e9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f62480cdc00 tx=0x7f6230009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:40.721 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.732+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f6248068fe0 con 0x7f624806c3d0 2024-03-27T06:22:40.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.972+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6248068840 con 0x7f624806c3d0 2024-03-27T06:22:40.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.974+0000 7f623f400640 1 -- 172.21.15.132:0/2358842507 <== mon.2 v2:172.21.15.134:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17374 (secure 0 0 0) 0x7f6238030810 con 0x7f624806c3d0 2024-03-27T06:22:40.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-27T06:22:40.964 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":44,"fsid":"3dfa4fc8-5726-4b88-a1bf-7da524e3644e","created":"2024-03-27T06:13:44.135271+0000","modified":"2024-03-27T06:21:29.204148+0000","last_up_change":"2024-03-27T06:13:54.436616+0000","last_in_change":"2024-03-27T06:13:46.613151+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T06:13:56.429180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-27T06:14:00.506476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-27T06:14:01.952491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":40,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"804650a8-bce8-4db0-87fc-a48dafbb9039","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6801","nonce":1970178383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6803","nonce":1970178383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6807","nonce":1970178383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":1970178383},{"type":"v1","addr":"172.21.15.132:6805","nonce":1970178383}]},"public_addr":"172.21.15.132:6801/1970178383","cluster_addr":"172.21.15.132:6803/1970178383","heartbeat_back_addr":"172.21.15.132:6807/1970178383","heartbeat_front_addr":"172.21.15.132:6805/1970178383","state":["exists","up"]},{"osd":1,"uuid":"7302bf98-cd95-401d-8a7f-c1b37532aa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6813","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6814","nonce":3659088073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6818","nonce":3659088073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6824","nonce":3659088073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6819","nonce":3659088073},{"type":"v1","addr":"172.21.15.132:6821","nonce":3659088073}]},"public_addr":"172.21.15.132:6814/3659088073","cluster_addr":"172.21.15.132:6818/3659088073","heartbeat_back_addr":"172.21.15.132:6824/3659088073","heartbeat_front_addr":"172.21.15.132:6821/3659088073","state":["exists","up"]},{"osd":2,"uuid":"d69ad874-437c-49b7-83ca-d7aef2e59047","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6823","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6825","nonce":1881883030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6827","nonce":1881883030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6831","nonce":1881883030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":1881883030},{"type":"v1","addr":"172.21.15.132:6829","nonce":1881883030}]},"public_addr":"172.21.15.132:6825/1881883030","cluster_addr":"172.21.15.132:6827/1881883030","heartbeat_back_addr":"172.21.15.132:6831/1881883030","heartbeat_front_addr":"172.21.15.132:6829/1881883030","state":["exists","up"]},{"osd":3,"uuid":"bb044ddc-5a7d-4afd-8ebb-39a2776d3046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6809","nonce":2165878941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6811","nonce":2165878941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6817","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6820","nonce":2165878941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2165878941},{"type":"v1","addr":"172.21.15.132:6815","nonce":2165878941}]},"public_addr":"172.21.15.132:6809/2165878941","cluster_addr":"172.21.15.132:6811/2165878941","heartbeat_back_addr":"172.21.15.132:6820/2165878941","heartbeat_front_addr":"172.21.15.132:6815/2165878941","state":["exists","up"]},{"osd":4,"uuid":"5fc09bbc-14fa-4638-8922-8f20d62654b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6817","nonce":2337158882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6819","nonce":2337158882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6823","nonce":2337158882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2337158882},{"type":"v1","addr":"172.21.15.134:6821","nonce":2337158882}]},"public_addr":"172.21.15.134:6817/2337158882","cluster_addr":"172.21.15.134:6819/2337158882","heartbeat_back_addr":"172.21.15.134:6823/2337158882","heartbeat_front_addr":"172.21.15.134:6821/2337158882","state":["exists","up"]},{"osd":5,"uuid":"ed2023b3-8f13-4157-ba83-e8245614800c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6809","nonce":172932420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6811","nonce":172932420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6815","nonce":172932420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":172932420},{"type":"v1","addr":"172.21.15.134:6813","nonce":172932420}]},"public_addr":"172.21.15.134:6809/172932420","cluster_addr":"172.21.15.134:6811/172932420","heartbeat_back_addr":"172.21.15.134:6815/172932420","heartbeat_front_addr":"172.21.15.134:6813/172932420","state":["exists","up"]},{"osd":6,"uuid":"e8b1479f-defc-4abe-81b7-ed751bce3325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6801","nonce":3892608742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6803","nonce":3892608742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6807","nonce":3892608742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3892608742},{"type":"v1","addr":"172.21.15.134:6805","nonce":3892608742}]},"public_addr":"172.21.15.134:6801/3892608742","cluster_addr":"172.21.15.134:6803/3892608742","heartbeat_back_addr":"172.21.15.134:6807/3892608742","heartbeat_front_addr":"172.21.15.134:6805/3892608742","state":["exists","up"]},{"osd":7,"uuid":"1ab5fc83-1a68-45ca-9b76-b0911127d9c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6825","nonce":789010038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6827","nonce":789010038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6831","nonce":789010038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":789010038},{"type":"v1","addr":"172.21.15.134:6829","nonce":789010038}]},"public_addr":"172.21.15.134:6825/789010038","cluster_addr":"172.21.15.134:6827/789010038","heartbeat_back_addr":"172.21.15.134:6831/789010038","heartbeat_front_addr":"172.21.15.134:6829/789010038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.110253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.877896+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.918020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:52.245213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.782921+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.791972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.775193+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T06:13:51.843143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6839/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6837/3802035585":"2024-03-28T06:19:09.174490+0000","172.21.15.134:6838/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6837/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.134:6836/2702381292":"2024-03-28T06:18:09.157115+0000","172.21.15.132:6836/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6834/3973814804":"2024-03-28T06:16:39.129873+0000","172.21.15.132:6835/1424864911":"2024-03-28T06:21:29.204044+0000","172.21.15.132:6837/690619582":"2024-03-28T06:15:19.112063+0000","172.21.15.132:6835/690619582":"2024-03-28T06:15:19.112063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T06:22:40.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 msgr2=0x7f622406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 0x7f622406e9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f62480cdc00 tx=0x7f6230009040 comp rx=0 tx=0).stop 2024-03-27T06:22:40.966 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 msgr2=0x7f62480cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:40.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f623802b9c0 tx=0x7f6238004730 comp rx=0 tx=0).stop 2024-03-27T06:22:40.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 shutdown_connections 2024-03-27T06:22:40.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f622406c530 0x7f622406e9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f624815bf30 0x7f62481f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6248154e00 0x7f624814c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 --2- 172.21.15.132:0/2358842507 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f624806c3d0 0x7f62480cdf30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:40.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 >> 172.21.15.132:0/2358842507 conn(0x7f62480d0150 msgr2=0x7f624814e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:40.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 shutdown_connections 2024-03-27T06:22:40.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:40.977+0000 7f624d800640 1 -- 172.21.15.132:0/2358842507 wait complete. 2024-03-27T06:22:40.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-27T06:22:41.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.197+0000 7fe88dc00640 1 Processor -- start 2024-03-27T06:22:41.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.197+0000 7fe88dc00640 1 -- start start 2024-03-27T06:22:41.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe88dc00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fe8880c8d70 0x7fe888164050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe88dc00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8880c6760 con 0x7fe8881529d0 2024-03-27T06:22:41.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe88dc00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8880c72a0 con 0x7fe8880c3b50 2024-03-27T06:22:41.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe88dc00640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fe8880c7050 con 0x7fe8880c8d70 2024-03-27T06:22:41.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe886a00640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe8881529d0 0x7fe88815c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:40400/0 (socket says 172.21.15.132:40400) 2024-03-27T06:22:41.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe886a00640 1 -- 172.21.15.132:0/1499352756 learned_addr learned my addr 172.21.15.132:0/1499352756 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:41.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe887400640 1 --2- 172.21.15.132:0/1499352756 >> v2:172.21.15.132:3300/0 conn(0x7fe8880c8d70 0x7fe888164050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe885600640 1 -- 172.21.15.132:0/1499352756 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411612714 0 0) 0x7fe8880c72a0 con 0x7fe8880c3b50 2024-03-27T06:22:41.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe885600640 1 -- 172.21.15.132:0/1499352756 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe86c003520 con 0x7fe8880c3b50 2024-03-27T06:22:41.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe887400640 1 -- 172.21.15.132:0/1499352756 >> v1:172.21.15.134:6789/0 conn(0x7fe8880c3b50 legacy=0x7fe8880c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe887400640 1 -- 172.21.15.132:0/1499352756 >> v1:172.21.15.132:6789/0 conn(0x7fe8881529d0 legacy=0x7fe88815c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.191 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.199+0000 7fe887400640 1 -- 172.21.15.132:0/1499352756 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8880c7580 con 0x7fe8880c8d70 2024-03-27T06:22:41.191 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.201+0000 7fe887400640 1 --2- 172.21.15.132:0/1499352756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c8d70 0x7fe888164050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe880009d90 tx=0x7fe88002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df84e7ac9efbf4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.191 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.201+0000 7fe885600640 1 -- 172.21.15.132:0/1499352756 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880039070 con 0x7fe8880c8d70 2024-03-27T06:22:41.191 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.201+0000 7fe885600640 1 -- 172.21.15.132:0/1499352756 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe88002cc60 con 0x7fe8880c8d70 2024-03-27T06:22:41.191 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.201+0000 7fe885600640 1 -- 172.21.15.132:0/1499352756 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880034680 con 0x7fe8880c8d70 2024-03-27T06:22:41.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 -- 172.21.15.132:0/1499352756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c8d70 msgr2=0x7fe888164050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1499352756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c8d70 0x7fe888164050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe880009d90 tx=0x7fe88002beb0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 -- 172.21.15.132:0/1499352756 shutdown_connections 2024-03-27T06:22:41.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1499352756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c8d70 0x7fe888164050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 -- 172.21.15.132:0/1499352756 >> 172.21.15.132:0/1499352756 conn(0x7fe8880cc650 msgr2=0x7fe8880cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:41.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 -- 172.21.15.132:0/1499352756 shutdown_connections 2024-03-27T06:22:41.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.202+0000 7fe88dc00640 1 -- 172.21.15.132:0/1499352756 wait complete. 2024-03-27T06:22:41.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.204+0000 7fe88dc00640 1 Processor -- start 2024-03-27T06:22:41.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.204+0000 7fe88dc00640 1 -- start start 2024-03-27T06:22:41.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 0x7fe88814b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 0x7fe8881f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe888152db0 con 0x7fe8880c3b50 2024-03-27T06:22:41.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880c7860 con 0x7fe8881529d0 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe88dc00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fe888067760 con 0x7fe8880c8d70 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 0x7fe88814b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37648/0 (socket says 172.21.15.132:37648) 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 0x7fe88814b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3301/0 says I am v2:172.21.15.132:50488/0 (socket says 172.21.15.132:50488) 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 -- 172.21.15.132:0/1129116230 learned_addr learned my addr 172.21.15.132:0/1129116230 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe887400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 0x7fe8881f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 -- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 msgr2=0x7fe88814b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 0x7fe88814b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 -- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 msgr2=0x7fe8881f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 0x7fe8881f0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe886a00640 1 -- 172.21.15.132:0/1129116230 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88814ca60 con 0x7fe8880c3b50 2024-03-27T06:22:41.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.205+0000 7fe887400640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 0x7fe8881f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:41.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe886a00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c3c0 tx=0x7fe87800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878017070 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe878005ce0 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878007310 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88814cd40 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.206+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88814d300 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.207+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe888068840 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.208+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fe87801a070 con 0x7fe8880c3b50 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.209+0000 7fe87ea00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 0x7fe86806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.209+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6168+0+0 (secure 0 0 0) 0x7fe87808ac40 con 0x7fe8880c3b50 2024-03-27T06:22:41.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.209+0000 7fe886000640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 0x7fe86806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.210+0000 7fe886000640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 0x7fe86806e9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe8881442b0 tx=0x7fe874009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.214+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fe8780589a0 con 0x7fe8880c3b50 2024-03-27T06:22:41.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.453+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe888150290 con 0x7fe8880c3b50 2024-03-27T06:22:41.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.457+0000 7fe87ea00640 1 -- 172.21.15.132:0/1129116230 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 161+0+0 (secure 0 0 0) 0x7fe87805cca0 con 0x7fe8880c3b50 2024-03-27T06:22:41.446 INFO:teuthology.orchestra.run.smithi132.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-27T06:22:41.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.459+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 msgr2=0x7fe86806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.459+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 0x7fe86806e9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe8881442b0 tx=0x7fe874009040 comp rx=0 tx=0).stop 2024-03-27T06:22:41.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.459+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 msgr2=0x7fe888073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.459+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c3c0 tx=0x7fe87800c880 comp rx=0 tx=0).stop 2024-03-27T06:22:41.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 shutdown_connections 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fe86806c530 0x7fe86806e9f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8881529d0 0x7fe8881f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fe8880c8d70 0x7fe88814b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 --2- 172.21.15.132:0/1129116230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8880c3b50 0x7fe888073260 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 >> 172.21.15.132:0/1129116230 conn(0x7fe8880cc650 msgr2=0x7fe888151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 shutdown_connections 2024-03-27T06:22:41.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.460+0000 7fe88dc00640 1 -- 172.21.15.132:0/1129116230 wait complete. 2024-03-27T06:22:41.469 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-27T06:22:41.479 INFO:tasks.ceph.mds.f.smithi134.stdout:starting mds.f at 2024-03-27T06:22:41.491 INFO:tasks.ceph.mds.b.smithi134.stdout:starting mds.b at 2024-03-27T06:22:41.530 INFO:tasks.ceph.mds.e.smithi132.stdout:starting mds.e at 2024-03-27T06:22:41.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.675+0000 7f5c50a00640 1 Processor -- start 2024-03-27T06:22:41.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.676+0000 7f5c50a00640 1 -- start start 2024-03-27T06:22:41.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c50a00640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c4c0c94c0 0x7f5c4c073800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c50a00640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4c15f920 con 0x7f5c4c1592d0 2024-03-27T06:22:41.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c50a00640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4c15fbe0 con 0x7f5c4c158ef0 2024-03-27T06:22:41.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c50a00640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f5c4c0c6ab0 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c49e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5c4c158ef0 0x7f5c4c0c6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:47506/0 (socket says 172.21.15.132:47506) 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.677+0000 7f5c49e00640 1 -- 172.21.15.132:0/2561138684 learned_addr learned my addr 172.21.15.132:0/2561138684 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c4a800640 1 --2- 172.21.15.132:0/2561138684 >> v2:172.21.15.134:3300/0 conn(0x7f5c4c0c94c0 0x7f5c4c073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958370789 0 0) 0x7f5c4c15fbe0 con 0x7f5c4c158ef0 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c2c003520 con 0x7f5c4c158ef0 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c4a800640 1 -- 172.21.15.132:0/2561138684 >> v1:172.21.15.134:6789/0 conn(0x7f5c4c158ef0 legacy=0x7f5c4c0c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3257116936 0 0) 0x7f5c2c003520 con 0x7f5c4c158ef0 2024-03-27T06:22:41.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c4a800640 1 -- 172.21.15.132:0/2561138684 >> v1:172.21.15.132:6789/0 conn(0x7f5c4c1592d0 legacy=0x7f5c4c0c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.678+0000 7f5c4a800640 1 -- 172.21.15.132:0/2561138684 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c0c0ec0 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.679+0000 7f5c4a800640 1 --2- 172.21.15.132:0/2561138684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c073800 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009d90 tx=0x7f5c4402bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6c8573ffabbdff server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.679+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44039070 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.679+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5c4402c830 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.679+0000 7f5c48a00640 1 -- 172.21.15.132:0/2561138684 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c001670 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 -- 172.21.15.132:0/2561138684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 msgr2=0x7f5c4c073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2561138684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c073800 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009d90 tx=0x7f5c4402bd00 comp rx=0 tx=0).stop 2024-03-27T06:22:41.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 -- 172.21.15.132:0/2561138684 shutdown_connections 2024-03-27T06:22:41.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2561138684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c073800 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 -- 172.21.15.132:0/2561138684 >> 172.21.15.132:0/2561138684 conn(0x7f5c4c0c89a0 msgr2=0x7f5c4c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:41.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 -- 172.21.15.132:0/2561138684 shutdown_connections 2024-03-27T06:22:41.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.680+0000 7f5c50a00640 1 -- 172.21.15.132:0/2561138684 wait complete. 2024-03-27T06:22:41.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.682+0000 7f5c50a00640 1 Processor -- start 2024-03-27T06:22:41.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.682+0000 7f5c50a00640 1 -- start start 2024-03-27T06:22:41.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 0x7f5c4c0725f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c4c1592d0 0x7f5c4c072b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c0c1480 con 0x7f5c4c158ef0 2024-03-27T06:22:41.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c0c17f0 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c50a00640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4c06c530 con 0x7f5c4c1592d0 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 0x7f5c4c0725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:54138/0 (socket says 172.21.15.132:54138) 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 -- 172.21.15.132:0/2048549440 learned_addr learned my addr 172.21.15.132:0/2048549440 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c4a800640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c4c1592d0 0x7f5c4c072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 -- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c4c1592d0 msgr2=0x7f5c4c072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c4c1592d0 0x7f5c4c072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 -- 172.21.15.132:0/2048549440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 msgr2=0x7f5c4c0725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:41.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 0x7f5c4c0725f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:41.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49e00640 1 -- 172.21.15.132:0/2048549440 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c14e340 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.684+0000 7f5c49400640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 0x7f5c4c0725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:41.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c49e00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c6f0 tx=0x7f5c3c00cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c005ad0 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5c3c005c70 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c007270 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4c0c6060 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.685+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c4c1550d0 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.686+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c4c068950 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.687+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f5c3c007410 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.688+0000 7f5c42000640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 0x7f5c2806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.688+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6426+0+0 (secure 0 0 0) 0x7f5c3c08b890 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.678 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.688+0000 7f5c49400640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 0x7f5c2806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:41.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.689+0000 7f5c49400640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 0x7f5c2806e9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0cb7c0 tx=0x7f5c34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:41.681 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.692+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f5c3c059460 con 0x7f5c4c0c94c0 2024-03-27T06:22:41.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:41.932+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5c4c0c8160 con 0x7f5c4c0c94c0 2024-03-27T06:22:42.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.462+0000 7f5c42000640 1 -- 172.21.15.132:0/2048549440 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) ==== 104+0+0 (secure 0 0 0) 0x7f5c3c05d760 con 0x7f5c4c0c94c0 2024-03-27T06:22:42.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 msgr2=0x7f5c2806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:42.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 0x7f5c2806e9f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0cb7c0 tx=0x7f5c34009040 comp rx=0 tx=0).stop 2024-03-27T06:22:42.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 msgr2=0x7f5c4c14d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:42.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c6f0 tx=0x7f5c3c00cbb0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 shutdown_connections 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f5c4c1592d0 0x7f5c4c072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c4c158ef0 0x7f5c4c0725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f5c2806c530 0x7f5c2806e9f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 --2- 172.21.15.132:0/2048549440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c4c0c94c0 0x7f5c4c14d9a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 >> 172.21.15.132:0/2048549440 conn(0x7f5c4c0c89a0 msgr2=0x7f5c4c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 shutdown_connections 2024-03-27T06:22:42.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.466+0000 7f5c50a00640 1 -- 172.21.15.132:0/2048549440 wait complete. 2024-03-27T06:22:42.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-27T06:22:42.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.704+0000 7fdb97200640 1 Processor -- start 2024-03-27T06:22:42.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.704+0000 7fdb97200640 1 -- start start 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb97200640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdb980c8b40 0x7fdb980c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb97200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb98073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb97200640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb980c8f20 con 0x7fdb98070e80 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb97200640 1 -- --> v2:172.21.15.132:3300/0 -- mon_getmap magic: 0 -- 0x7fdb980c77e0 con 0x7fdb980c8b40 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb97200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdb98157fe0 con 0x7fdb980c7390 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdb980c8b40 0x7fdb980c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 --2- >> v2:172.21.15.132:3300/0 conn(0x7fdb980c8b40 0x7fdb980c7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44238/0 (socket says 172.21.15.132:44238) 2024-03-27T06:22:42.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 -- 172.21.15.132:0/625468299 learned_addr learned my addr 172.21.15.132:0/625468299 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:42.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb96800640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb98073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 -- 172.21.15.132:0/625468299 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 msgr2=0x7fdb98073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:42.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb98073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.706+0000 7fdb95e00640 1 -- 172.21.15.132:0/625468299 >> v1:172.21.15.134:6789/0 conn(0x7fdb98070e80 legacy=0x7fdb981574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:42.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.707+0000 7fdb95e00640 1 -- 172.21.15.132:0/625468299 --> v2:172.21.15.132:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb981582c0 con 0x7fdb980c8b40 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.708+0000 7fdb95e00640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb980c7c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009e50 tx=0x7fdb8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e5d7bd0fd6d14b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.708+0000 7fdb94a00640 1 -- 172.21.15.132:0/625468299 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8802ca80 con 0x7fdb980c8b40 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.708+0000 7fdb94a00640 1 -- 172.21.15.132:0/625468299 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdb8802cc20 con 0x7fdb980c8b40 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.708+0000 7fdb94a00640 1 -- 172.21.15.132:0/625468299 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb880343f0 con 0x7fdb980c8b40 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 -- 172.21.15.132:0/625468299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 msgr2=0x7fdb980c7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb980c7c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009e50 tx=0x7fdb8802bee0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 -- 172.21.15.132:0/625468299 shutdown_connections 2024-03-27T06:22:42.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb98073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 --2- 172.21.15.132:0/625468299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb980c7c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 -- 172.21.15.132:0/625468299 >> 172.21.15.132:0/625468299 conn(0x7fdb980676a0 msgr2=0x7fdb9806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:42.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 -- 172.21.15.132:0/625468299 shutdown_connections 2024-03-27T06:22:42.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.709+0000 7fdb97200640 1 -- 172.21.15.132:0/625468299 wait complete. 2024-03-27T06:22:42.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.711+0000 7fdb97200640 1 Processor -- start 2024-03-27T06:22:42.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.711+0000 7fdb97200640 1 -- start start 2024-03-27T06:22:42.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 0x7fdb98144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb981531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdb980c6b90 con 0x7fdb980c8b40 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdb980c68b0 con 0x7fdb98070e80 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb97200640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7fdb980c6e70 con 0x7fdb980c7390 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb96800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb96800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44250/0 (socket says 172.21.15.132:44250) 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb95e00640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 0x7fdb98144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb95400640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb981531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.712+0000 7fdb96800640 1 -- 172.21.15.132:0/3840187124 learned_addr learned my addr 172.21.15.132:0/3840187124 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 -- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 msgr2=0x7fdb981531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb981531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 -- 172.21.15.132:0/3840187124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 msgr2=0x7fdb9814d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 -- 172.21.15.132:0/3840187124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9814bed0 con 0x7fdb98070e80 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb96800640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb95e00640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 0x7fdb98144a90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdb88032000 tx=0x7fdb88004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:42.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.713+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88043020 con 0x7fdb98070e80 2024-03-27T06:22:42.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.714+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdb88004250 con 0x7fdb98070e80 2024-03-27T06:22:42.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.714+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88009510 con 0x7fdb98070e80 2024-03-27T06:22:42.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.714+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb981590f0 con 0x7fdb98070e80 2024-03-27T06:22:42.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.714+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9814c140 con 0x7fdb98070e80 2024-03-27T06:22:42.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.715+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb98068730 con 0x7fdb98070e80 2024-03-27T06:22:42.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.716+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7fdb88046020 con 0x7fdb98070e80 2024-03-27T06:22:42.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.717+0000 7fdb8e000640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 0x7fdb7806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:42.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.717+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6426+0+0 (secure 0 0 0) 0x7fdb880adfe0 con 0x7fdb98070e80 2024-03-27T06:22:42.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.717+0000 7fdb95400640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 0x7fdb7806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:42.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.718+0000 7fdb95400640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 0x7fdb7806e9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdb980722e0 tx=0x7fdb84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:42.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.720+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7fdb8804d030 con 0x7fdb98070e80 2024-03-27T06:22:42.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:42.926+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdb980b1e90 con 0x7fdb98070e80 2024-03-27T06:22:43.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.697+0000 7fdb8e000640 1 -- 172.21.15.132:0/3840187124 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) ==== 181+0+0 (secure 0 0 0) 0x7fdb981590f0 con 0x7fdb98070e80 2024-03-27T06:22:43.687 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'cephfs_metadata' removed 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.701+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 msgr2=0x7fdb7806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.701+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 0x7fdb7806e9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdb980722e0 tx=0x7fdb84009040 comp rx=0 tx=0).stop 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.701+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 msgr2=0x7fdb98144a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.701+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 0x7fdb98144a90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdb88032000 tx=0x7fdb88004a40 comp rx=0 tx=0).stop 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 shutdown_connections 2024-03-27T06:22:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb980c8b40 0x7fdb9814d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7fdb980c7390 0x7fdb981531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7fdb7806c530 0x7fdb7806e9f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 --2- 172.21.15.132:0/3840187124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb98070e80 0x7fdb98144a90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 >> 172.21.15.132:0/3840187124 conn(0x7fdb980676a0 msgr2=0x7fdb98156090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 shutdown_connections 2024-03-27T06:22:43.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.702+0000 7fdb97200640 1 -- 172.21.15.132:0/3840187124 wait complete. 2024-03-27T06:22:43.720 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-27T06:22:43.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.928+0000 7f215d000640 1 Processor -- start 2024-03-27T06:22:43.918 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.928+0000 7f215d000640 1 -- start start 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.929+0000 7f215d000640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f215815f9a0 0x7f215806c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.929+0000 7f215d000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f215806cc40 0x7f21580c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.929+0000 7f215d000640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21580c1ae0 con 0x7f21581590f0 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.929+0000 7f215d000640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 -- 0x7f21581594d0 con 0x7f215815f9a0 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.929+0000 7f215d000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f21580c1890 con 0x7f215806cc40 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f214d400640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f215815f9a0 0x7f215806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f2156800640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f215806cc40 0x7f21580c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f2155e00640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f21581590f0 0x7f21580c7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.132:52886/0 (socket says 172.21.15.132:52886) 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f2155e00640 1 -- 172.21.15.132:0/1322804514 learned_addr learned my addr 172.21.15.132:0/1322804514 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f2155400640 1 -- 172.21.15.132:0/1322804514 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365912622 0 0) 0x7f21580c1ae0 con 0x7f21581590f0 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f214d400640 1 -- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f215806cc40 msgr2=0x7f21580c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f214d400640 1 --2- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f215806cc40 0x7f21580c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f214d400640 1 -- 172.21.15.132:0/1322804514 >> v1:172.21.15.134:6789/0 conn(0x7f21581590f0 legacy=0x7f21580c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.930+0000 7f214d400640 1 -- 172.21.15.132:0/1322804514 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21580c1dc0 con 0x7f215815f9a0 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.931+0000 7f214d400640 1 --2- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215806c700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21400097a0 tx=0x7f214002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13b9241970fdd1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f2155400640 1 -- 172.21.15.132:0/1322804514 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140034070 con 0x7f215815f9a0 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f2155400640 1 -- 172.21.15.132:0/1322804514 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2140030610 con 0x7f215815f9a0 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f2155400640 1 -- 172.21.15.132:0/1322804514 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140030a00 con 0x7f215815f9a0 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 -- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 msgr2=0x7f215806c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 --2- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215806c700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21400097a0 tx=0x7f214002b960 comp rx=0 tx=0).stop 2024-03-27T06:22:43.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 -- 172.21.15.132:0/1322804514 shutdown_connections 2024-03-27T06:22:43.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 --2- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f215806cc40 0x7f21580c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 --2- 172.21.15.132:0/1322804514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215806c700 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.932+0000 7f215d000640 1 -- 172.21.15.132:0/1322804514 >> 172.21.15.132:0/1322804514 conn(0x7f21580c8a80 msgr2=0x7f21580c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:43.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.933+0000 7f215d000640 1 -- 172.21.15.132:0/1322804514 shutdown_connections 2024-03-27T06:22:43.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.933+0000 7f215d000640 1 -- 172.21.15.132:0/1322804514 wait complete. 2024-03-27T06:22:43.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.934+0000 7f215d000640 1 Processor -- start 2024-03-27T06:22:43.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.934+0000 7f215d000640 1 -- start start 2024-03-27T06:22:43.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 0x7f21581560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f21581590f0 0x7f215814db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21580c2380 con 0x7f215806cc40 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21580c29d0 con 0x7f215815f9a0 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f215d000640 1 -- --> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] -- mon_getmap magic: 0 -- 0x7f21580c2660 con 0x7f21581590f0 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f2155e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 0x7f21581560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f2156800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f2155e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 0x7f21581560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44256/0 (socket says 172.21.15.132:44256) 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f214d400640 1 --2- >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f21581590f0 0x7f215814db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f2156800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.132:33924/0 (socket says 172.21.15.132:33924) 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.935+0000 7f2155e00640 1 -- 172.21.15.132:0/1530677145 learned_addr learned my addr 172.21.15.132:0/1530677145 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 -- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f21581590f0 msgr2=0x7f215814db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f21581590f0 0x7f215814db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 -- 172.21.15.132:0/1530677145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 msgr2=0x7f21581560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:43.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 0x7f21581560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 -- 172.21.15.132:0/1530677145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21580cb380 con 0x7f215815f9a0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f2156800640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f215000e7c0 tx=0x7f215000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.936+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150017070 con 0x7f215815f9a0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.937+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f215000c730 con 0x7f215815f9a0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.937+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21581f8580 con 0x7f215815f9a0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.937+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215000cb80 con 0x7f215815f9a0 2024-03-27T06:22:43.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.937+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581f8820 con 0x7f215815f9a0 2024-03-27T06:22:43.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.938+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2158068840 con 0x7f215815f9a0 2024-03-27T06:22:43.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.939+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) ==== 90856+0+0 (secure 0 0 0) 0x7f215001a070 con 0x7f215815f9a0 2024-03-27T06:22:43.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.940+0000 7f214ea00640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 0x7f213406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T06:22:43.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.940+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5988+0+0 (secure 0 0 0) 0x7f215008b270 con 0x7f215815f9a0 2024-03-27T06:22:43.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.940+0000 7f2155e00640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 0x7f213406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T06:22:43.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.940+0000 7f2155e00640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 0x7f213406ea20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f21480012e0 tx=0x7f21480017e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T06:22:43.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:43.944+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186417 (secure 0 0 0) 0x7f2150058ff0 con 0x7f215815f9a0 2024-03-27T06:22:44.133 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.143+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f21580cadc0 con 0x7f215815f9a0 2024-03-27T06:22:44.702 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.712+0000 7f214ea00640 1 -- 172.21.15.132:0/1530677145 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) ==== 169+0+0 (secure 0 0 0) 0x7f215005d2f0 con 0x7f215815f9a0 2024-03-27T06:22:44.702 INFO:teuthology.orchestra.run.smithi132.stderr:pool 'cephfs_data' removed 2024-03-27T06:22:44.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 msgr2=0x7f213406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:44.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 0x7f213406ea20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f21480012e0 tx=0x7f21480017e0 comp rx=0 tx=0).stop 2024-03-27T06:22:44.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 msgr2=0x7f215814e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T06:22:44.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f215000e7c0 tx=0x7f215000ec80 comp rx=0 tx=0).stop 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 shutdown_connections 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:6832/2615516388,v1:172.21.15.134:6833/2615516388] conn(0x7f213406c560 0x7f213406ea20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215815f9a0 0x7f215814e050 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] conn(0x7f21581590f0 0x7f215814db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 --2- 172.21.15.132:0/1530677145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f215806cc40 0x7f21581560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.717+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 >> 172.21.15.132:0/1530677145 conn(0x7f21580c8a80 msgr2=0x7f215806d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.718+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 shutdown_connections 2024-03-27T06:22:44.708 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-27T06:22:44.718+0000 7f215d000640 1 -- 172.21.15.132:0/1530677145 wait complete. 2024-03-27T06:22:44.739 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-27T06:22:44.739 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-27T06:22:44.739 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.769 INFO:tasks.ceph.mds.a:Stopped 2024-03-27T06:22:44.769 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-27T06:22:44.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.829 INFO:tasks.ceph.mds.c:Stopped 2024-03-27T06:22:44.829 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-03-27T06:22:44.829 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.888 INFO:tasks.ceph.mds.e:Stopped 2024-03-27T06:22:44.888 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-27T06:22:44.888 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.898 INFO:tasks.ceph.mds.b:Stopped 2024-03-27T06:22:44.898 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-27T06:22:44.898 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.948 INFO:tasks.ceph.mds.d:Stopped 2024-03-27T06:22:44.949 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-03-27T06:22:44.949 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:44.998 INFO:tasks.ceph.mds.f:Stopped 2024-03-27T06:22:44.998 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-27T06:22:44.999 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-27T06:22:44.999 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:45.085 INFO:tasks.ceph.osd.0:Stopped 2024-03-27T06:22:45.085 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T06:22:45.085 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:45.169 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T06:22:45.170 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T06:22:45.170 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:45.240 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T06:22:45.241 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-27T06:22:45.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:46.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-27T06:22:48.757 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-03-27T06:22:48.768+0000 7f82aa041640 -1 calc_pg_upmaps abort due to max <= 0 2024-03-27T06:22:51.343 INFO:tasks.ceph.osd.3:Stopped 2024-03-27T06:22:51.343 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-27T06:22:51.343 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:51.373 INFO:tasks.ceph.osd.4:Stopped 2024-03-27T06:22:51.373 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-27T06:22:51.373 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:51.449 INFO:tasks.ceph.osd.5:Stopped 2024-03-27T06:22:51.450 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-27T06:22:51.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:51.516 INFO:tasks.ceph.osd.6:Stopped 2024-03-27T06:22:51.517 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-27T06:22:51.517 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:52.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-27T06:22:52.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-27T06:22:57.619 INFO:tasks.ceph.osd.7:Stopped 2024-03-27T06:22:57.619 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-27T06:22:57.620 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-27T06:22:57.620 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:57.687 INFO:tasks.ceph.mgr.y:Stopped 2024-03-27T06:22:57.688 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-27T06:22:57.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:57.717 INFO:tasks.ceph.mgr.x:Stopped 2024-03-27T06:22:57.717 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-27T06:22:57.717 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-27T06:22:57.717 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:57.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2024-03-27T06:22:57.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-03-27T06:22:57.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-03-27T06:22:57.767 INFO:tasks.ceph.mon.a:Stopped 2024-03-27T06:22:57.768 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-27T06:22:57.768 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:57.781 INFO:tasks.ceph.mon.b:Stopped 2024-03-27T06:22:57.782 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-27T06:22:57.782 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T06:22:57.836 INFO:tasks.ceph.mon.c:Stopped 2024-03-27T06:22:57.836 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-27T06:22:57.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | head -n 1 2024-03-27T06:22:57.957 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:22:57.957 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-27T06:22:58.359 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:22:58.360 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-27T06:22:58.527 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:22:58.527 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-27T06:22:58.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:22:58.667 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-27T06:22:58.866 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:22:58.866 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-27T06:22:59.088 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:22:59.089 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-27T06:22:59.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:22:59.192 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-27T06:22:59.255 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:22:59.255 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-27T06:22:59.317 INFO:tasks.ceph:Archiving mon data... 2024-03-27T06:22:59.319 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/data/mon.a.tgz 2024-03-27T06:22:59.319 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-27T06:22:59.376 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.bfi85yZRFM 2024-03-27T06:22:59.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bfi85yZRFM 2024-03-27T06:22:59.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.bfi85yZRFM 2024-03-27T06:22:59.694 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.bfi85yZRFM is 286KB 2024-03-27T06:22:59.801 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.bfi85yZRFM 2024-03-27T06:22:59.856 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/data/mon.b.tgz 2024-03-27T06:22:59.857 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-27T06:22:59.874 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.0Kquw1x3eN 2024-03-27T06:22:59.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0Kquw1x3eN 2024-03-27T06:23:00.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.0Kquw1x3eN 2024-03-27T06:23:00.096 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.0Kquw1x3eN is 286KB 2024-03-27T06:23:00.194 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.0Kquw1x3eN 2024-03-27T06:23:00.209 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/data/mon.c.tgz 2024-03-27T06:23:00.209 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-27T06:23:00.263 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.qLypfQoRBc 2024-03-27T06:23:00.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.qLypfQoRBc 2024-03-27T06:23:00.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.qLypfQoRBc 2024-03-27T06:23:00.466 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.qLypfQoRBc is 286KB 2024-03-27T06:23:00.564 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.qLypfQoRBc 2024-03-27T06:23:00.578 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-27T06:23:00.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T06:23:00.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T06:23:00.735 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T06:23:00.750 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T06:23:00.750 INFO:tasks.ceph:Archiving crash dumps... 2024-03-27T06:23:00.752 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi132/crash 2024-03-27T06:23:00.753 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T06:23:00.848 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi134/crash 2024-03-27T06:23:00.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T06:23:00.874 INFO:tasks.ceph:Shutting down logrotate 2024-03-27T06:23:00.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:23:00.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-27T06:23:01.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-27T06:23:01.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-27T06:23:01.134 INFO:tasks.ceph:Compressing logs... 2024-03-27T06:23:01.134 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T06:23:01.141 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T06:23:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-27T06:23:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38309.log 2024-03-27T06:23:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T06:23:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38309.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-27T06:23:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38269.log 2024-03-27T06:23:01.207 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38309.log.gz 2024-03-27T06:23:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-27T06:23:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38269.log.gz 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38349.log 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38469.log: /var/log/ceph/ceph-client.admin.38349.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-27T06:23:01.208 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38349.log.gz/var/log/ceph/ceph-osd.7.log: 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38389.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38229.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1%/var/log/ceph/ceph-mds.f.log: -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-27T06:23:01.209 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-03-27T06:23:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-27T06:23:01.212 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-27T06:23:01.212 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38429.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-03-27T06:23:01.212 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-03-27T06:23:01.224 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-27T06:23:01.225 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38509.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-03-27T06:23:01.228 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45826.log 2024-03-27T06:23:01.229 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51509.log 2024-03-27T06:23:01.230 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45826.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-27T06:23:01.232 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-03-27T06:23:01.232 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51449.log 2024-03-27T06:23:01.233 INFO:teuthology.orchestra.run.smithi132.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51509.log.gz 2024-03-27T06:23:01.234 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46564.log 2024-03-27T06:23:01.234 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-27T06:23:01.235 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51449.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51449.log.gz 2024-03-27T06:23:01.235 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43943.log 2024-03-27T06:23:01.236 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2024-03-27T06:23:01.236 INFO:teuthology.orchestra.run.smithi132.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46564.log.gz 2024-03-27T06:23:01.237 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-03-27T06:23:01.237 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43943.log.gz 2024-03-27T06:23:01.238 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43622.log 2024-03-27T06:23:01.238 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2024-03-27T06:23:01.240 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50536.log 2024-03-27T06:23:01.240 INFO:teuthology.orchestra.run.smithi132.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2024-03-27T06:23:01.240 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.0.43622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43622.log.gz 2024-03-27T06:23:01.241 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2024-03-27T06:23:01.242 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50536.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50536.log.gz 2024-03-27T06:23:01.242 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52839.log 2024-03-27T06:23:01.244 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52377.log 2024-03-27T06:23:01.244 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2024-03-27T06:23:01.245 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50969.log 2024-03-27T06:23:01.245 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52839.log.gz 2024-03-27T06:23:01.246 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45258.log 2024-03-27T06:23:01.246 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52377.log.gz 2024-03-27T06:23:01.247 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mds.d.log: 93.7%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-27T06:23:01.247 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50969.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50969.log.gz 2024-03-27T06:23:01.247 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41529.log 2024-03-27T06:23:01.248 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46175.log 2024-03-27T06:23:01.248 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45258.log.gz 2024-03-27T06:23:01.249 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2024-03-27T06:23:01.249 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41529.log.gz 2024-03-27T06:23:01.250 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46175.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46175.log.gz 2024-03-27T06:23:01.251 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50410.log 2024-03-27T06:23:01.252 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50200.log 2024-03-27T06:23:01.252 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2024-03-27T06:23:01.253 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50116.log 2024-03-27T06:23:01.253 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50410.log.gz 2024-03-27T06:23:01.254 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50200.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50200.log.gz 2024-03-27T06:23:01.254 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52923.log 2024-03-27T06:23:01.256 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52881.log 2024-03-27T06:23:01.256 INFO:teuthology.orchestra.run.smithi132.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50116.log.gz 2024-03-27T06:23:01.258 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-03-27T06:23:01.258 INFO:teuthology.orchestra.run.smithi132.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52923.log.gz 2024-03-27T06:23:01.258 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52881.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52881.log.gz 2024-03-27T06:23:01.258 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53162.log 2024-03-27T06:23:01.259 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44046.log 2024-03-27T06:23:01.259 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-03-27T06:23:01.260 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53162.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53162.log.gz 2024-03-27T06:23:01.261 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44267.log 2024-03-27T06:23:01.262 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53288.log 2024-03-27T06:23:01.262 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44046.log.gz 2024-03-27T06:23:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2024-03-27T06:23:01.263 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44267.log.gz 2024-03-27T06:23:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53288.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53288.log.gz 2024-03-27T06:23:01.264 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44140.log 2024-03-27T06:23:01.265 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52461.log 2024-03-27T06:23:01.265 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2024-03-27T06:23:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2024-03-27T06:23:01.266 INFO:teuthology.orchestra.run.smithi132.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44140.log.gz 2024-03-27T06:23:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42503.log 2024-03-27T06:23:01.267 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52461.log.gz 2024-03-27T06:23:01.268 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41867.log 2024-03-27T06:23:01.269 INFO:teuthology.orchestra.run.smithi132.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2024-03-27T06:23:01.271 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50326.log 2024-03-27T06:23:01.271 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42503.log.gz 2024-03-27T06:23:01.271 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41867.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41867.log.gz 2024-03-27T06:23:01.271 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52251.log 2024-03-27T06:23:01.273 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43287.log 2024-03-27T06:23:01.273 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50326.log.gz 2024-03-27T06:23:01.273 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-03-27T06:23:01.274 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52251.log.gz 2024-03-27T06:23:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48994.log 2024-03-27T06:23:01.276 INFO:teuthology.orchestra.run.smithi132.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43287.log.gz 2024-03-27T06:23:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-03-27T06:23:01.276 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-03-27T06:23:01.277 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52755.log 2024-03-27T06:23:01.277 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48994.log.gz 2024-03-27T06:23:01.278 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51874.log 2024-03-27T06:23:01.278 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-03-27T06:23:01.279 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2024-03-27T06:23:01.279 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52755.log.gz 2024-03-27T06:23:01.280 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36736.log 2024-03-27T06:23:01.280 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51874.log.gz 2024-03-27T06:23:01.281 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48671.log 2024-03-27T06:23:01.281 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2024-03-27T06:23:01.282 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.36736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44762.log 2024-03-27T06:23:01.282 INFO:teuthology.orchestra.run.smithi132.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36736.log.gz 2024-03-27T06:23:01.283 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48671.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.42924.log -- replaced with /var/log/ceph/ceph-client.admin.48671.log.gz 2024-03-27T06:23:01.283 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-27T06:23:01.286 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48907.log 2024-03-27T06:23:01.286 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44762.log.gz 2024-03-27T06:23:01.286 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-03-27T06:23:01.286 INFO:teuthology.orchestra.run.smithi132.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42924.log.gz 2024-03-27T06:23:01.287 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48907.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48907.log.gz 2024-03-27T06:23:01.287 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44981.log 2024-03-27T06:23:01.288 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42206.log 2024-03-27T06:23:01.288 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-03-27T06:23:01.289 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46073.log 2024-03-27T06:23:01.289 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44981.log.gz 2024-03-27T06:23:01.290 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42206.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42206.log.gz 2024-03-27T06:23:01.290 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-03-27T06:23:01.291 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-03-27T06:23:01.291 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46073.log.gz 2024-03-27T06:23:01.292 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-03-27T06:23:01.292 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz 2024-03-27T06:23:01.293 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-03-27T06:23:01.294 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-03-27T06:23:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-27T06:23:01.295 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-03-27T06:23:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51738.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-03-27T06:23:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45300.log 2024-03-27T06:23:01.297 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 2024-03-27T06:23:01.297 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-03-27T06:23:01.298 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45300.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45300.log.gz 2024-03-27T06:23:01.298 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51263.log 2024-03-27T06:23:01.299 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42671.log 2024-03-27T06:23:01.299 INFO:teuthology.orchestra.run.smithi132.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2024-03-27T06:23:01.300 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51263.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51263.log.gz 2024-03-27T06:23:01.300 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51677.log 2024-03-27T06:23:01.301 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42671.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.42671.log.gz 2024-03-27T06:23:01.302 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2024-03-27T06:23:01.303 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42461.log 2024-03-27T06:23:01.303 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51677.log.gz 2024-03-27T06:23:01.304 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45198.log 2024-03-27T06:23:01.304 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2024-03-27T06:23:01.305 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44889.log 2024-03-27T06:23:01.305 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42461.log.gz 2024-03-27T06:23:01.306 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49929.log 2024-03-27T06:23:01.306 INFO:teuthology.orchestra.run.smithi132.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45198.log.gz 2024-03-27T06:23:01.307 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44889.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44889.log.gz 2024-03-27T06:23:01.307 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46704.log 2024-03-27T06:23:01.308 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49215.log 2024-03-27T06:23:01.308 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49929.log.gz 2024-03-27T06:23:01.309 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43050.log 2024-03-27T06:23:01.310 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46704.log.gz 2024-03-27T06:23:01.310 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-03-27T06:23:01.311 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49215.log.gz 2024-03-27T06:23:01.311 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48952.log 2024-03-27T06:23:01.312 INFO:teuthology.orchestra.run.smithi132.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.43050.log.gz 2024-03-27T06:23:01.312 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-27T06:23:01.313 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-03-27T06:23:01.313 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48952.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2024-03-27T06:23:01.313 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52671.log 2024-03-27T06:23:01.314 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48952.log.gz 2024-03-27T06:23:01.314 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-03-27T06:23:01.314 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-03-27T06:23:01.315 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42411.log 2024-03-27T06:23:01.316 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52671.log.gz 2024-03-27T06:23:01.316 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46115.log 2024-03-27T06:23:01.317 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-03-27T06:23:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42411.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42411.log.gz 2024-03-27T06:23:01.318 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2024-03-27T06:23:01.319 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50242.log 2024-03-27T06:23:01.319 INFO:teuthology.orchestra.run.smithi132.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46115.log.gz 2024-03-27T06:23:01.320 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52209.log 87.9% 2024-03-27T06:23:01.320 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2024-03-27T06:23:01.321 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50242.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.50242.log.gz 2024-03-27T06:23:01.321 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46396.log 2024-03-27T06:23:01.322 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52209.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T06:23:01.323 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52209.log.gz 2024-03-27T06:23:01.323 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53433.log 2024-03-27T06:23:01.323 INFO:teuthology.orchestra.run.smithi132.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46396.log.gz 2024-03-27T06:23:01.324 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48823.log 2024-03-27T06:23:01.325 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53433.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-27T06:23:01.325 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53433.log.gz 2024-03-27T06:23:01.326 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49120.log 2024-03-27T06:23:01.326 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52335.log 2024-03-27T06:23:01.327 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48823.log.gz 2024-03-27T06:23:01.327 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45023.log 2024-03-27T06:23:01.328 INFO:teuthology.orchestra.run.smithi132.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49120.log.gz 2024-03-27T06:23:01.328 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53330.log 2024-03-27T06:23:01.329 INFO:teuthology.orchestra.run.smithi132.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52335.log.gz 2024-03-27T06:23:01.330 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50452.log 2024-03-27T06:23:01.330 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45023.log.gz 2024-03-27T06:23:01.331 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41487.log 2024-03-27T06:23:01.331 INFO:teuthology.orchestra.run.smithi132.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53330.log.gz 2024-03-27T06:23:01.332 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43008.log 2024-03-27T06:23:01.332 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50452.log.gz 2024-03-27T06:23:01.333 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41879.log 2024-03-27T06:23:01.333 INFO:teuthology.orchestra.run.smithi132.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41487.log.gz 2024-03-27T06:23:01.334 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50687.log 2024-03-27T06:23:01.334 INFO:teuthology.orchestra.run.smithi132.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43008.log.gz 2024-03-27T06:23:01.335 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37094.log 2024-03-27T06:23:01.335 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41879.log.gz 2024-03-27T06:23:01.336 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51305.log 2024-03-27T06:23:01.336 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50687.log.gz 2024-03-27T06:23:01.337 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.37094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44435.log 2024-03-27T06:23:01.337 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37094.log.gz 2024-03-27T06:23:01.338 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42215.log 2024-03-27T06:23:01.338 INFO:teuthology.orchestra.run.smithi132.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51305.log.gz 2024-03-27T06:23:01.339 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44091.log 2024-03-27T06:23:01.339 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44435.log.gz 2024-03-27T06:23:01.340 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42215.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.42215.log.gz 2024-03-27T06:23:01.340 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46606.log 2024-03-27T06:23:01.341 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42210.log 2024-03-27T06:23:01.341 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44091.log.gz 2024-03-27T06:23:01.342 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48461.log 2024-03-27T06:23:01.342 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46606.log.gz 2024-03-27T06:23:01.343 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53007.log 2024-03-27T06:23:01.343 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42210.log.gz 2024-03-27T06:23:01.344 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40576.log 2024-03-27T06:23:01.344 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48461.log.gz 2024-03-27T06:23:01.345 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52293.log 2024-03-27T06:23:01.345 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53007.log.gz 2024-03-27T06:23:01.347 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51593.log 2024-03-27T06:23:01.347 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40576.log.gz 2024-03-27T06:23:01.347 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51011.log 2024-03-27T06:23:01.348 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52293.log.gz 2024-03-27T06:23:01.348 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43901.log 2024-03-27T06:23:01.349 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51593.log.gz 2024-03-27T06:23:01.349 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-27T06:23:01.350 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51011.log.gz 2024-03-27T06:23:01.350 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53246.log 2024-03-27T06:23:01.351 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43901.log.gz 2024-03-27T06:23:01.352 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-27T06:23:01.352 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-27T06:23:01.353 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49336.log 2024-03-27T06:23:01.353 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53246.log.gz 2024-03-27T06:23:01.354 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51053.log 2024-03-27T06:23:01.355 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46351.log 2024-03-27T06:23:01.355 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49336.log.gz 2024-03-27T06:23:01.356 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2024-03-27T06:23:01.356 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51053.log.gz 2024-03-27T06:23:01.357 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44939.log 2024-03-27T06:23:01.357 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46351.log.gz 2024-03-27T06:23:01.358 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2024-03-27T06:23:01.358 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2024-03-27T06:23:01.359 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44939.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44939.log.gz 2024-03-27T06:23:01.359 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-27T06:23:01.360 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41697.log 2024-03-27T06:23:01.361 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2024-03-27T06:23:01.361 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42797.log 2024-03-27T06:23:01.362 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-27T06:23:01.362 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42629.log 2024-03-27T06:23:01.363 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41697.log.gz 2024-03-27T06:23:01.364 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41872.log 2024-03-27T06:23:01.364 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42797.log.gz 2024-03-27T06:23:01.365 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52587.log 2024-03-27T06:23:01.365 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42629.log.gz 2024-03-27T06:23:01.366 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42966.log 2024-03-27T06:23:01.366 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41872.log.gz 2024-03-27T06:23:01.367 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50368.log 2024-03-27T06:23:01.367 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52587.log.gz 2024-03-27T06:23:01.368 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49162.log 2024-03-27T06:23:01.368 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42966.log.gz 2024-03-27T06:23:01.369 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42755.log 2024-03-27T06:23:01.369 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50368.log.gz 2024-03-27T06:23:01.370 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52713.log 2024-03-27T06:23:01.370 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49162.log.gz 2024-03-27T06:23:01.371 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48731.log 2024-03-27T06:23:01.371 INFO:teuthology.orchestra.run.smithi132.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42755.log.gz 2024-03-27T06:23:01.372 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 2024-03-27T06:23:01.373 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz 2024-03-27T06:23:01.374 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48731.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30711.log 2024-03-27T06:23:01.374 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48731.log.gz 2024-03-27T06:23:01.375 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45068.log 2024-03-27T06:23:01.375 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2024-03-27T06:23:01.375 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.tmp-client.admin.30711.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30711.log.gz 2024-03-27T06:23:01.376 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53475.log 2024-03-27T06:23:01.377 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-03-27T06:23:01.377 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45068.log.gz 2024-03-27T06:23:01.378 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49971.log 2024-03-27T06:23:01.378 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53475.log.gz 2024-03-27T06:23:01.379 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2024-03-27T06:23:01.380 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-03-27T06:23:01.380 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49971.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49971.log.gz 2024-03-27T06:23:01.380 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46438.log 2024-03-27T06:23:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-27T06:23:01.381 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-03-27T06:23:01.382 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46438.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.46438.log.gz --verbose 2024-03-27T06:23:01.382 INFO:teuthology.orchestra.run.smithi132.stderr: -- /var/log/ceph/ceph-client.admin.42839.log 2024-03-27T06:23:01.383 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41875.log 2024-03-27T06:23:01.384 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42839.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42839.log.gz 2024-03-27T06:23:01.385 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44846.log 2024-03-27T06:23:01.386 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49036.log 2024-03-27T06:23:01.387 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41875.log: /var/log/ceph/ceph-client.admin.44846.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41875.log.gz 2024-03-27T06:23:01.387 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44846.log.gz 2024-03-27T06:23:01.387 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44636.log 2024-03-27T06:23:01.388 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51095.log 2024-03-27T06:23:01.389 INFO:teuthology.orchestra.run.smithi132.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49036.log.gz 2024-03-27T06:23:01.390 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44636.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44636.log.gz 2024-03-27T06:23:01.390 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-03-27T06:23:01.391 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2024-03-27T06:23:01.391 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51095.log.gz 2024-03-27T06:23:01.392 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42212.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-03-27T06:23:01.392 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log 2024-03-27T06:23:01.393 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-03-27T06:23:01.394 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52797.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2024-03-27T06:23:01.395 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz 2024-03-27T06:23:01.395 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-03-27T06:23:01.396 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40691.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-03-27T06:23:01.396 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53390.log 2024-03-27T06:23:01.397 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51635.log 2024-03-27T06:23:01.398 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-03-27T06:23:01.398 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50494.log 2024-03-27T06:23:01.399 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53390.log.gz 2024-03-27T06:23:01.399 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51635.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51635.log.gz 2024-03-27T06:23:01.400 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51800.log 2024-03-27T06:23:01.401 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44309.log 2024-03-27T06:23:01.401 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50494.log.gz 2024-03-27T06:23:01.402 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51800.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51800.log.gz 2024-03-27T06:23:01.402 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45110.log 2024-03-27T06:23:01.403 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-03-27T06:23:01.404 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44309.log.gz 2024-03-27T06:23:01.405 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49887.log 2024-03-27T06:23:01.405 INFO:teuthology.orchestra.run.smithi132.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45110.log.gz 2024-03-27T06:23:01.406 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49078.log 2024-03-27T06:23:01.406 INFO:teuthology.orchestra.run.smithi132.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-03-27T06:23:01.407 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49887.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49887.log.gz 2024-03-27T06:23:01.407 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50596.log 2024-03-27T06:23:01.408 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49078.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49078.log.gz 2024-03-27T06:23:01.408 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36991.log 2024-03-27T06:23:01.409 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48865.log 2024-03-27T06:23:01.410 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50596.log.gz 2024-03-27T06:23:01.411 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.36991.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-27T06:23:01.411 INFO:teuthology.orchestra.run.smithi132.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36991.log.gz 2024-03-27T06:23:01.412 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42587.log 2024-03-27T06:23:01.412 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48865.log.gz 2024-03-27T06:23:01.413 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-03-27T06:23:01.415 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-03-27T06:23:01.415 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42587.log.gz 2024-03-27T06:23:01.416 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.49845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-27T06:23:01.416 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-03-27T06:23:01.417 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50158.log 2024-03-27T06:23:01.418 INFO:teuthology.orchestra.run.smithi132.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-03-27T06:23:01.419 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-27T06:23:01.419 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-27T06:23:01.420 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2024-03-27T06:23:01.420 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50158.log.gz 2024-03-27T06:23:01.421 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2024-03-27T06:23:01.423 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51179.log 2024-03-27T06:23:01.423 INFO:teuthology.orchestra.run.smithi132.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2024-03-27T06:23:01.424 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.47170.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2024-03-27T06:23:01.425 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2024-03-27T06:23:01.426 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51179.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2024-03-27T06:23:01.426 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2024-03-27T06:23:01.427 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48629.log 2024-03-27T06:23:01.427 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2024-03-27T06:23:01.428 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41865.log 2024-03-27T06:23:01.429 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2024-03-27T06:23:01.430 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50927.log 2024-03-27T06:23:01.430 INFO:teuthology.orchestra.run.smithi132.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48629.log.gz 2024-03-27T06:23:01.431 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41865.log: 94.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-27T06:23:01.431 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42199.log 2024-03-27T06:23:01.431 INFO:teuthology.orchestra.run.smithi132.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41865.log.gz 2024-03-27T06:23:01.432 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47011.log 2024-03-27T06:23:01.432 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50927.log.gz 2024-03-27T06:23:01.433 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44225.log 2024-03-27T06:23:01.434 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42199.log.gz 2024-03-27T06:23:01.435 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.47011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42545.log 2024-03-27T06:23:01.435 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47011.log.gz 2024-03-27T06:23:01.436 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43185.log 2024-03-27T06:23:01.436 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44225.log.gz 2024-03-27T06:23:01.437 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51551.log 2024-03-27T06:23:01.438 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42545.log.gz 2024-03-27T06:23:01.439 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46217.log 2024-03-27T06:23:01.439 INFO:teuthology.orchestra.run.smithi132.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43185.log.gz 2024-03-27T06:23:01.440 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51407.log 2024-03-27T06:23:01.440 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51551.log.gz 2024-03-27T06:23:01.441 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46217.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-27T06:23:01.441 INFO:teuthology.orchestra.run.smithi132.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46217.log.gz 2024-03-27T06:23:01.442 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52545.log 2024-03-27T06:23:01.443 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51407.log.gz 2024-03-27T06:23:01.443 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48545.log 2024-03-27T06:23:01.445 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50284.log 2024-03-27T06:23:01.445 INFO:teuthology.orchestra.run.smithi132.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52545.log.gz 2024-03-27T06:23:01.447 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42713.log 2024-03-27T06:23:01.447 INFO:teuthology.orchestra.run.smithi132.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48545.log.gz 2024-03-27T06:23:01.448 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.50284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44804.log 2024-03-27T06:23:01.448 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50284.log.gz 2024-03-27T06:23:01.449 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51221.log 2024-03-27T06:23:01.449 INFO:teuthology.orchestra.run.smithi132.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42713.log.gz 2024-03-27T06:23:01.451 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44804.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44804.log.gz 2024-03-27T06:23:01.451 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53204.log 2024-03-27T06:23:01.452 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51221.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51221.log.gz 2024-03-27T06:23:01.452 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-27T06:23:01.454 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.53204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53122.log 2024-03-27T06:23:01.454 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53204.log.gz 2024-03-27T06:23:01.456 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.53122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45468.log 2024-03-27T06:23:01.457 INFO:teuthology.orchestra.run.smithi132.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-27T06:23:01.457 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53122.log.gz 2024-03-27T06:23:01.458 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-27T06:23:01.459 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44477.log 2024-03-27T06:23:01.460 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45468.log: 88.5%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.45468.log.gz 2024-03-27T06:23:01.461 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-27T06:23:01.462 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-27T06:23:01.462 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44477.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44477.log.gz 2024-03-27T06:23:01.463 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50074.log 2024-03-27T06:23:01.465 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.50074.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50074.log.gz 2024-03-27T06:23:01.466 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48587.log 2024-03-27T06:23:01.468 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.48587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-03-27T06:23:01.470 INFO:teuthology.orchestra.run.smithi132.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46522.log 2024-03-27T06:23:01.471 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48587.log.gz 2024-03-27T06:23:01.472 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40995.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-03-27T06:23:01.473 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49511.log 2024-03-27T06:23:01.476 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46522.log: /var/log/ceph/ceph-client.admin.49511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41877.log 2024-03-27T06:23:01.477 INFO:teuthology.orchestra.run.smithi132.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49511.log.gz 2024-03-27T06:23:01.478 INFO:teuthology.orchestra.run.smithi132.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2024-03-27T06:23:01.478 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42214.log 2024-03-27T06:23:01.482 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41877.log: /var/log/ceph/ceph-client.admin.42214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44182.log 2024-03-27T06:23:01.483 INFO:teuthology.orchestra.run.smithi132.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41877.log.gz 2024-03-27T06:23:01.484 INFO:teuthology.orchestra.run.smithi132.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42214.log.gz 2024-03-27T06:23:01.485 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51916.log 2024-03-27T06:23:01.487 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44720.log 2024-03-27T06:23:01.490 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51916.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51916.log.gz 2024-03-27T06:23:01.491 INFO:teuthology.orchestra.run.smithi132.stderr: 88.3%gzip -5 --verbose 92.2% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-27T06:23:01.492 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44720.log: -- /var/log/ceph/ceph-client.admin.52629.log 2024-03-27T06:23:01.492 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44182.log.gz 2024-03-27T06:23:01.493 INFO:teuthology.orchestra.run.smithi132.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44720.log.gz 2024-03-27T06:23:01.493 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41571.log 2024-03-27T06:23:01.495 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52629.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2024-03-27T06:23:01.495 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-03-27T06:23:01.496 INFO:teuthology.orchestra.run.smithi132.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-03-27T06:23:01.498 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41871.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-03-27T06:23:01.499 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41089.log 2024-03-27T06:23:01.500 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45406.log 2024-03-27T06:23:01.501 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41089.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41089.log.gz 2024-03-27T06:23:01.503 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.45406.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45406.log.gz 2024-03-27T06:23:01.503 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-27T06:23:01.505 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2024-03-27T06:23:01.506 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41655.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-03-27T06:23:01.507 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41235.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2024-03-27T06:23:01.508 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43775.log 2024-03-27T06:23:01.509 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52965.log 2024-03-27T06:23:01.511 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.43775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43775.log.gz 2024-03-27T06:23:01.512 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.52965.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52965.log.gz 2024-03-27T06:23:01.513 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51137.log 2024-03-27T06:23:01.514 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46480.log 2024-03-27T06:23:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.51137.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51137.log.gz 2024-03-27T06:23:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.46480.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46480.log.gz 2024-03-27T06:23:01.517 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2024-03-27T06:23:01.519 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-03-27T06:23:01.520 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41133.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2024-03-27T06:23:01.521 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41361.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-03-27T06:23:01.522 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-27T06:23:01.523 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50030.log 2024-03-27T06:23:01.526 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.50030.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50030.log.gz 2024-03-27T06:23:01.527 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-03-27T06:23:01.527 INFO:teuthology.orchestra.run.smithi132.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-27T06:23:01.530 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42197.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-03-27T06:23:01.530 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44393.log 2024-03-27T06:23:01.531 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42881.log 2024-03-27T06:23:01.533 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.44393.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44393.log.gz 2024-03-27T06:23:01.534 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42881.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42881.log.gz 2024-03-27T06:23:01.618 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-27T06:23:01.666 INFO:teuthology.orchestra.run.smithi132.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-27T06:23:01.723 INFO:teuthology.orchestra.run.smithi132.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-27T06:23:02.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-27T06:23:02.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-03-27T06:23:02.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-03-27T06:23:04.665 INFO:teuthology.orchestra.run.smithi132.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-27T06:23:05.000 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-27T06:23:05.528 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-27T06:23:05.558 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-27T06:23:05.726 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-27T06:23:06.481 INFO:teuthology.orchestra.run.smithi132.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-27T06:23:07.012 INFO:teuthology.orchestra.run.smithi132.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-27T06:23:07.015 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-27T06:23:07.015 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m5.827s 2024-03-27T06:23:07.016 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m17.561s 2024-03-27T06:23:07.016 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m1.986s 2024-03-27T06:23:07.045 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-27T06:23:18.396 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-27T06:23:20.936 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-27T06:23:20.938 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-27T06:23:20.938 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m19.750s 2024-03-27T06:23:20.939 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m54.368s 2024-03-27T06:23:20.939 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m1.672s 2024-03-27T06:23:20.939 INFO:tasks.ceph:Archiving logs... 2024-03-27T06:23:20.939 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi132/log 2024-03-27T06:23:20.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T06:23:23.944 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi134/log 2024-03-27T06:23:23.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T06:23:31.606 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-27T06:23:31.618 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-27T06:23:31.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T06:23:31.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T06:23:31.767 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-27T06:23:31.777 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-27T06:23:31.777 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T06:23:31.784 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T06:23:31.801 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-27T06:23:31.806 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T06:23:31.806 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-27T06:23:31.807 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +89us[ +86us] +/- 95ms 2024-03-27T06:23:31.807 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1050us[-1053us] +/- 66ms 2024-03-27T06:23:31.807 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +987us[ +984us] +/- 64ms 2024-03-27T06:23:31.807 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:23:31.832 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-27T06:23:31.841 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T06:23:31.842 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-27T06:23:31.842 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +114us[ +114us] +/- 95ms 2024-03-27T06:23:31.842 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1002us[-1008us] +/- 66ms 2024-03-27T06:23:31.842 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1004us[+1004us] +/- 64ms 2024-03-27T06:23:31.842 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T06:23:31.843 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-27T06:23:31.853 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-27T06:23:31.853 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-27T06:23:31.863 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T06:23:32.085 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T06:23:32.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T06:23:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:23:32.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T06:23:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:23:32.293 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-27T06:23:32.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-27T06:23:32.368 INFO:teuthology.task.internal:Duration was 1203.608598 seconds 2024-03-27T06:23:32.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-27T06:23:32.390 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-27T06:23:32.390 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T06:23:32.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T06:23:32.445 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T06:23:32.547 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T06:23:32.995 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-27T06:23:32.995 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-03-27T06:23:32.996 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T06:23:33.058 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-03-27T06:23:33.059 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T06:23:33.089 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-27T06:23:33.089 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T06:23:33.097 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T06:23:33.185 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-27T06:23:33.186 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T06:23:33.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T06:23:33.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-27T06:23:33.385 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-27T06:23:33.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T06:23:33.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T06:23:33.505 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-27T06:23:33.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T06:23:33.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T06:23:33.571 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-03-27T06:23:33.631 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-03-27T06:23:33.684 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T06:23:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:23:33.729 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T06:23:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T06:23:33.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-27T06:23:33.755 INFO:teuthology.task.internal:Transferring archived files... 2024-03-27T06:23:33.756 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi132 2024-03-27T06:23:33.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T06:23:33.874 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580/remote/smithi134 2024-03-27T06:23:33.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T06:23:33.912 INFO:teuthology.task.internal:Removing archive directory... 2024-03-27T06:23:33.912 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T06:23:33.920 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T06:23:33.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-27T06:23:33.980 INFO:teuthology.task.internal:Not uploading archives. 2024-03-27T06:23:33.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-27T06:23:34.000 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-27T06:23:34.000 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T06:23:34.008 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T06:23:34.023 INFO:teuthology.orchestra.run.smithi134.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 06:23 /home/ubuntu/cephtest 2024-03-27T06:23:34.056 INFO:teuthology.orchestra.run.smithi132.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 06:23 /home/ubuntu/cephtest 2024-03-27T06:23:34.060 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-27T06:23:34.070 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-27T06:23:34.091 INFO:teuthology.nuke:Checking targets against current locks 2024-03-27T06:23:34.111 DEBUG:teuthology.nuke:shortname: smithi132 2024-03-27T06:23:34.112 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T06:23:34.133 DEBUG:teuthology.nuke:shortname: smithi134 2024-03-27T06:23:34.134 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T06:23:34.137 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 05:54:25.389416', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlC8b1Xb9cMub9U5aw87k8Okmsf9s9uwlDii5o/NuWDK9hG4g1SdEHKoONFWWqP1X63Vx69lC4WeZYVst7rfoI='} 2024-03-27T06:23:34.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-27_05:27:11-fs-wip-rishabh-testing-20240326.131558-testing-default-smithi/7625580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 05:54:25.387398', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODXlBN77fBLoFuEWP82nY+5NNH+IIIOpJ3kkasqacKGnvNe23UPM31XQt76Zglt/8nIOdWIZthrOyatmFeLyeI='} 2024-03-27T06:23:34.193 INFO:teuthology.orchestra.console.smithi132:Power off 2024-03-27T06:23:34.194 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T06:23:34.213 INFO:teuthology.orchestra.console.smithi134:Power off 2024-03-27T06:23:34.214 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T06:23:34.228 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-03-27T06:23:34.228 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:34.241 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-03-27T06:23:34.242 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:34.257 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-27T06:23:34.269 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-27T06:23:38.259 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:38.270 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:38.370 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-27T06:23:38.380 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-27T06:23:42.374 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:42.383 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T06:23:42.485 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-03-27T06:23:42.492 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-03-27T06:23:42.586 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-03-27T06:23:42.593 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-03-27T06:23:42.701 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1203.6085977554321 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi132 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=155268c4e432a12433aa833f174f9fe3b1016ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b631062c33ed425682ae5636c619af39 status: fail success: false 2024-03-27T06:23:42.701 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T06:23:42.790 INFO:teuthology.run:FAIL